gemma: pkg/imports/fd.go annotate

annotate pkg/imports/fd.go @ 5560:f2204f91d286

Join the log lines of imports to the log exports to recover data from them. Used in SR export to extract information that where in the meta json but now are only found in the log.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Wed, 09 Feb 2022 18:34:40 +0100
parents	3bab0e19f08b
children	6270951dda28

rev	line source
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
5275 73563c4bba5b Slight improvement of log messages for fd import. wilde@azure1.rgb.intevation.de parents: 5270 diff changeset	7 // Copyright (C) 2018, 2019, 2020 by via donau
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	13
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	14 package imports
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	15
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 import (
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17 "context"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 "database/sql"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "encoding/json"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "fmt"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	21 "io"
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4389 diff changeset	22 "strings"
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	23 "time"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	24
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	25 "gemma.intevation.de/gemma/pkg/common"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	26 "gemma.intevation.de/gemma/pkg/pgxutils"
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	27 "gemma.intevation.de/gemma/pkg/wfs"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	28 )
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	30 // FairwayDimension is an import job to import
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	31 // the fairway dimensions in form of polygon geometries
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32 // and attribute data from a WFS service.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33 type FairwayDimension struct {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	34 // URL the GetCapabilities URL of the WFS service.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	35 URL string `json:"url"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	36 // FeatureType selects the feature type of the WFS service.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	37 FeatureType string `json:"feature-type"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	38 SortBy string `json:"sort-by"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	39 LOS int `json:"los"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	40 MinWidth int `json:"min-width"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	41 MaxWidth int `json:"max-width"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	42 Depth int `json:"depth"`
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43 SourceOrganization string `json:"source-organization"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	44 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	45 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	46 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	47 Password string `json:"password,omitempty"`
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	48 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	49
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4832 diff changeset	50 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5277 diff changeset	51 func (fd *FairwayDimension) Description([]string) (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	52 return strings.Join([]string{
f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	53 fd.URL,
f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	54 fd.FeatureType,
f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	55 fmt.Sprintf("LOS%d", fd.LOS),
f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	56 }, "\|"), nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4389 diff changeset	57 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4389 diff changeset	58
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	59 type fdTime struct{ time.Time }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	60
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	61 var guessFDTime = common.TimeParser([]string{
1940 e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	62 "20060102",
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	63 "2006",
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	64 "",
4074 eb2f949ddfa2 Rename TimeGuesser to TimeParser. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4071 diff changeset	65 }).Parse
1940 e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	66
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	67 func (fdt *fdTime) UnmarshalJSON(data []byte) error {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	68 var s string
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 if err := json.Unmarshal(data, &s); err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 return err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 }
1940 e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	72 t, err := guessFDTime(s)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 if err == nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 *fdt = fdTime{t}
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	75 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	76 return err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	78
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 // FDJobKind is the import queue type identifier.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	80 const FDJobKind JobKind = "fd"
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	81
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	82 type fdJobCreator struct{}
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	83
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 func init() {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 RegisterJobCreator(FDJobKind, fdJobCreator{})
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 func (fdJobCreator) Description() string { return "fairway dimension" }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	90 func (fdJobCreator) AutoAccept() bool { return false }
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2171 diff changeset	92 func (fdJobCreator) Create() Job { return new(FairwayDimension) }
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	93
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2923 diff changeset	94 func (fdJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2923 diff changeset	95 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2923 diff changeset	96 {"fairway_dimensions"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2923 diff changeset	97 {"level_of_service"},
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	98 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	99 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	100
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	101 // StageDone replaces fairway dimensions with those in the staging area
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	102 func (fdJobCreator) StageDone(
b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	103 ctx context.Context,
b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	104 tx *sql.Tx,
b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	105 id int64,
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5032 diff changeset	106 _ Feedback,
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	107 ) error {
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	108 // Delete the old features.
5026 d02f0c5c60b3 Simplify deletion of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5024 diff changeset	109 if _, err := tx.ExecContext(ctx, deleteFairwayDimensionSQL, id); err != nil {
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	110 return err
9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	111 }
9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	112
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	113 _, err := tx.ExecContext(ctx, fdStageDoneSQL, id)
b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	114 return err
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	115 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	116
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	117 // CleanUp for fairway dimension imports is a NOP.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	118 func (*FairwayDimension) CleanUp() error { return nil }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	119
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	120 type fairwayDimensionProperties struct {
1940 e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	121 HydroSorDat *fdTime `json:"hydro_sordat"`
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	122 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	123
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	124 type fdSummary struct {
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	125 Lat float64 `json:"lat"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	126 Lon float64 `json:"lon"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	127 ID int64 `json:"id"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	128 }
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	129
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	130 const (
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	131 fdStageDoneSQL = `
b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	132 UPDATE waterway.fairway_dimensions SET staging_done = true
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	133 WHERE id IN (
2171 543b5d071a69 Fairway dimension imports: Only delete features with the same level of services as the imported. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 SELECT key FROM import.track_imports
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	135 WHERE import_id = $1 AND
2171 543b5d071a69 Fairway dimension imports: Only delete features with the same level of services as the imported. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 relation = 'waterway.fairway_dimensions'::regclass)`
1890 b1b0db195cc5 Put fairway dimension import to staging area. Raimund Renkert <raimund.renkert@intevation.de> parents: 1841 diff changeset	137
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	138 deleteFairwayDimensionSQL = `
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	139 -- Delete entries to be replaced by those in staging area
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	140 DELETE FROM waterway.fairway_dimensions
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	141 WHERE id IN (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	142 SELECT key FROM import.track_imports
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	143 WHERE import_id = $1
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	144 AND relation = 'waterway.fairway_dimensions'::regclass
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	145 AND deletion)
5026 d02f0c5c60b3 Simplify deletion of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5024 diff changeset	146 `
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	147 // Temporary table to collect IDs of unchanged entries
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	148 tmpTableSQL = `
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	149 CREATE TEMP TABLE unchanged (id int PRIMARY KEY) ON COMMIT DROP
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	150 `
2171 543b5d071a69 Fairway dimension imports: Only delete features with the same level of services as the imported. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	151 // The ST_MakeValid and ST_Buffer below are a workarround to
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 // avoid errors due to reprojection.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153 insertFairwayDimensionSQL = `
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	155 SELECT users.current_user_area_utm() AS a
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	156 ),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	157 g AS (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	158 SELECT ST_Multi(ST_CollectionExtract(ST_MakeValid(ST_Transform(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	159 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	160 OR ST_Covers((SELECT a FROM resp),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	161 ST_Transform(new_fd, (SELECT ST_SRID(a) FROM resp)))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	162 THEN new_fd
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	163 ELSE ST_Intersection(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	164 (SELECT ST_Buffer(a, -0.0001) FROM resp),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	165 ST_MakeValid(ST_Transform(new_fd, (SELECT ST_SRID(a) FROM resp))))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	166 END,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	167 4326)), 3)) AS new_fd
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	168 FROM ST_GeomFromWKB($1, $2::integer) AS new_fd (new_fd)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	169 WHERE pg_has_role('sys_admin', 'MEMBER')
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	170 OR ST_Intersects((SELECT a FROM resp),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	171 ST_MakeValid(ST_Transform(new_fd, (SELECT ST_SRID(a) FROM resp))))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	172 ),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	173 not_new AS (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	174 -- Collect IDs of unchanged entries in temp table
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	175 INSERT INTO unchanged
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	176 SELECT id
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	177 FROM g, waterway.fairway_dimensions
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	178 WHERE staging_done
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	179 AND validity @> current_timestamp
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	180 AND (area, level_of_service,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	181 min_width, max_width, min_depth, source_organization
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	182 ) IS NOT DISTINCT FROM (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	183 new_fd, $3, $4, $5, $6, $8)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	184 -- Return something if a duplicate in the data source is encountered
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	185 ON CONFLICT (id) DO UPDATE SET id = EXCLUDED.id
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	186 RETURNING 1
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	187 )
2923 9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	188 INSERT INTO waterway.fairway_dimensions (
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	189 area,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	190 level_of_service,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	191 min_width,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	192 max_width,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	193 min_depth,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	194 date_info,
9bd98daf0b0f Let sys_admin import fairway dimensions without restrictions Tom Gottfried <tom@intevation.de> parents: 2922 diff changeset	195 source_organization)
5031 8c590ef35280 Improve feedback if geometry cannot be stored Tom Gottfried <tom@intevation.de> parents: 5030 diff changeset	196 SELECT
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	197 new_fd, $3, $4, $5, $6, $7, $8
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	198 FROM g
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	199 WHERE NOT EXISTS(SELECT 1 FROM not_new)
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	200 RETURNING id,
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	201 ST_X(ST_Centroid(area::geometry)),
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	202 ST_Y(ST_Centroid(area::geometry))
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	203 `
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	204 // Fetch IDs of entries removed from data source
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	205 selectOldSQL = `
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	206 WITH resp AS (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	207 SELECT users.current_user_area_utm() AS a
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	208 )
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	209 SELECT id FROM waterway.fairway_dimensions
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	210 WHERE staging_done
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	211 AND validity @> current_timestamp
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	212 AND level_of_service = $1
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	213 AND (pg_has_role('sys_admin', 'MEMBER')
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	214 OR ST_Covers((SELECT a FROM resp),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	215 ST_Transform(CAST(area AS geometry), (SELECT ST_SRID(a) FROM resp))))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	216 AND id NOT IN (SELECT id FROM unchanged)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	217 `
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	218 invalidateFairwayDimensionSQL = `
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	219 WITH track AS (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	220 -- Mark entry for deletion that has been removed from the data source
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	221 INSERT INTO import.track_imports (import_id, deletion, relation, key)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	222 VALUES($1, true, 'waterway.fairway_dimensions', $2)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	223 )
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	224 -- Insert historic version with respective validity
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	225 INSERT INTO waterway.fairway_dimensions (
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	226 area,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	227 validity,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	228 level_of_service,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	229 min_width,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	230 max_width,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	231 min_depth,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	232 date_info,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	233 source_organization)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	234 SELECT
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	235 area,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	236 tstzrange(lower(validity), current_timestamp),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	237 level_of_service,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	238 min_width,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	239 max_width,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	240 min_depth,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	241 date_info,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	242 source_organization
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	243 FROM waterway.fairway_dimensions
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	244 WHERE id = $2
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	245 RETURNING id,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	246 ST_X(ST_Centroid(area::geometry)),
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	247 ST_Y(ST_Centroid(area::geometry))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	248 `
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	249 )
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	250
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	251 // Do executes the actual fairway dimension import.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	252 func (fd *FairwayDimension) Do(
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	253 ctx context.Context,
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	254 importID int64,
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	255 conn *sql.Conn,
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	256 feedback Feedback,
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	257 ) (interface{}, error) {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	258
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 start := time.Now()
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	260
1994 a7c4005b723f Improve some import log messages Tom Gottfried <tom@intevation.de> parents: 1941 diff changeset	261 feedback.Info("Import fairway dimensions")
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 feedback.Info("Loading capabilities from %s", fd.URL)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264 caps, err := wfs.GetCapabilities(fd.URL)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	265 if err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	266 feedback.Error("Loading capabilities failed: %v", err)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	267 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	268 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	269
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	270 ft := caps.FindFeatureType(fd.FeatureType)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	271 if ft == nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	272 return nil, fmt.Errorf("unknown feature type '%s'", fd.FeatureType)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	273 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	274
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	275 feedback.Info("Found feature type '%s'", fd.FeatureType)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	276
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	277 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	278 if err != nil {
4832 5062ccb2381d Unified error message on "Unsupported CRS" Sascha Wilde <wilde@intevation.de> parents: 4799 diff changeset	279 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	280 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	281 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	282
2721 56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2719 diff changeset	283 dl, err := wfs.GetFeatures(caps, fd.FeatureType, fd.SortBy)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	284 if err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	285 feedback.Error("Cannot create GetFeature URLs. %v", err)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 tx, err := conn.BeginTx(ctx, nil)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 if err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 defer tx.Rollback()
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	294
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	295 if _, err := tx.ExecContext(ctx, tmpTableSQL); err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	296 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	297 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	298
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	299 insertStmt, err := tx.PrepareContext(ctx, insertFairwayDimensionSQL)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	300 if err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	301 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	302 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	303 defer insertStmt.Close()
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	304
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	305 invalidateStmt, err := tx.PrepareContext(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	306 ctx, invalidateFairwayDimensionSQL)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	307 if err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	308 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	309 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	310 defer invalidateStmt.Close()
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	311
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	312 savepoint := Savepoint(ctx, tx, "feature")
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	313
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	314 var (
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	315 unsupported = stringCounter{}
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	316 missingProperties int
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	317 badProperties int
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	318 features int
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	319 outside int
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	320 fds []fdSummary
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	321 )
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	322
2729 7cb027be277d WFS downloader: Really use username and password from parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2723 diff changeset	323 if err := dl.Download(fd.User, fd.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	324 feedback.Info("Get features from: '%s'", url)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	325 rfc, err := wfs.ParseRawFeatureCollection(r)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	326 if err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	327 return fmt.Errorf("parsing GetFeature document failed: %v", err)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	328 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	329 if rfc.CRS != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	330 crsName := rfc.CRS.Properties.Name
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	331 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	332 feedback.Error("Unsupported CRS: %d", crsName)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	333 return err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	334 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	335 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	336
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	337 // No features -> ignore.
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	338 if rfc.Features == nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	339 return nil
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	340 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	341
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	342 feedback.Info("Using EPSG: %d", epsg)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	343
5023 776db50c01f0 Fix repairing geometries Tom Gottfried <tom@intevation.de> parents: 4852 diff changeset	344 feedback.Info(
776db50c01f0 Fix repairing geometries Tom Gottfried <tom@intevation.de> parents: 4852 diff changeset	345 "Found %d features in data source", len(rfc.Features))
776db50c01f0 Fix repairing geometries Tom Gottfried <tom@intevation.de> parents: 4852 diff changeset	346
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	347 features:
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	348 for _, feature := range rfc.Features {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	349 if feature.Geometry.Coordinates == nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	350 missingProperties++
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	351 continue
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	352 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	353
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	354 var props fairwayDimensionProperties
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	355
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	356 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
1941 d4249b07e719 Fairway dimension: log bad properties. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1940 diff changeset	357 feedback.Warn("bad property: %v", err)
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	358 badProperties++
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	359 continue
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	360 }
1940 e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	361 var dateInfo time.Time
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	362 if props.HydroSorDat == nil \|\| props.HydroSorDat.IsZero() {
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	363 dateInfo = start
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	364 } else {
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	365 dateInfo = (*props.HydroSorDat).Time
e6d851f53c99 Fairway dimension import: Made date parsing more tolerant. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1934 diff changeset	366 }
5270 41a67619c170 Added multi geometry support for some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5262 diff changeset	367
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	368 var polys multiPolygonSlice
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	369
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	370 switch feature.Geometry.Type {
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	371 case "Polygon":
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	372 var p polygonSlice
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	373 if err := json.Unmarshal(*feature.Geometry.Coordinates, &p); err != nil {
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	374 return err
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	375 }
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	376 polys = multiPolygonSlice{p}
5270 41a67619c170 Added multi geometry support for some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5262 diff changeset	377
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	378 case "MultiPolygon":
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	379 if err := json.Unmarshal(*feature.Geometry.Coordinates, &polys); err != nil {
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	380 return err
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	381 }
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	382 default:
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	383 unsupported[feature.Geometry.Type]++
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	384 continue features
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	385 }
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	386
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	387 // Store the features.
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	388 storePolygons:
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	389 for _, p := range polys {
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	390 var fdid int64
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	391 var lat, lon float64
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	392 switch err := savepoint(func() error {
569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	393 return insertStmt.QueryRowContext(
2922 372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	394 ctx,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	395 p.asWKB(),
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	396 epsg,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	397 fd.LOS,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	398 fd.MinWidth,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	399 fd.MaxWidth,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	400 fd.Depth,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	401 dateInfo,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	402 fd.SourceOrganization,
372dc8382389 Fairway dimension import: continue if a feature fails to be inserted Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	403 ).Scan(&fdid, &lat, &lon)
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	404 }); {
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	405 case err == sql.ErrNoRows:
ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	406 outside++
4389 5e38667f740c Use stretches as areas of responsibility. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	407 // ignore -> filtered by responsibility area (stretches)
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	408 continue storePolygons
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	409 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	410 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	411 continue storePolygons
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	412 }
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	413 // Store for potential later removal.
5271 569aaba23e20 Simplified FD multi geometry code and don't give up if one part failed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5270 diff changeset	414 if err := track(
5270 41a67619c170 Added multi geometry support for some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5262 diff changeset	415 ctx, tx, importID, "waterway.fairway_dimensions", fdid,
41a67619c170 Added multi geometry support for some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5262 diff changeset	416 ); err != nil {
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	417 return err
9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	418 }
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	419 fds = append(fds, fdSummary{ID: fdid, Lat: lat, Lon: lon})
1893 9beff970b107 Track fairway dimension import and delete old data on staging done. Raimund Renkert <raimund.renkert@intevation.de> parents: 1890 diff changeset	420
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	421 features++
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	422 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	423 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	424 return nil
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	425 }); err != nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	426 return nil, err
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	427 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	428
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	429 if badProperties > 0 {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	430 feedback.Warn("Bad properties: %d", badProperties)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	431 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	432
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	433 if missingProperties > 0 {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	434 feedback.Warn("Missing properties: %d", missingProperties)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	435 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	436
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	437 if len(unsupported) != 0 {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	438 feedback.Warn("Unsupported types found: %s", unsupported)
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	439 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	440
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	441 if outside > 0 {
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	442 feedback.Info(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	443 "Features outside responsibility area or unchanged: %d", outside)
1934 ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	444 }
ec7360afd5e1 Fairway dimension import: Dont break if there are features outside the responsibility areas. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1898 diff changeset	445
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	446 if features == 0 {
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	447 feedback.Info("No new features found")
5275 73563c4bba5b Slight improvement of log messages for fd import. wilde@azure1.rgb.intevation.de parents: 5270 diff changeset	448 } else {
73563c4bba5b Slight improvement of log messages for fd import. wilde@azure1.rgb.intevation.de parents: 5270 diff changeset	449 feedback.Info("Stored %d features", features)
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	450 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	451
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	452 // Invalidate features that have been removed from data source
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	453 res, err := tx.QueryContext(ctx, selectOldSQL, fd.LOS)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	454 if err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	455 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	456 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	457 defer res.Close()
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	458 var oldIDs []int64
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	459 for res.Next() {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	460 var oldID int64
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	461 if err := res.Scan(&oldID); err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	462 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	463 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	464 oldIDs = append(oldIDs, oldID)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	465 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	466 if err := res.Err(); err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	467 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	468 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	469
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	470 if features == 0 && len(oldIDs) == 0 {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	471 return nil, UnchangedError("Nothing changed")
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	472 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	473
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	474 if len(oldIDs) > 0 {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	475 feedback.Info(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	476 "Number of features removed from data source: %d", len(oldIDs))
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	477
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	478 var old int
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	479 for _, oldID := range oldIDs {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	480 var fdid int64
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	481 var lat, lon float64
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	482 if err := savepoint(func() error {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	483 return invalidateStmt.QueryRowContext(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	484 ctx,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	485 importID,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	486 oldID,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	487 ).Scan(&fdid, &lat, &lon)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	488 }); err != nil {
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5277 diff changeset	489 feedback.Error(pgxutils.ReadableError{Err: err}.Error()+
5275 73563c4bba5b Slight improvement of log messages for fd import. wilde@azure1.rgb.intevation.de parents: 5270 diff changeset	490 "- while tracking invalidation of: %d", oldID)
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	491 continue
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	492 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	493 fds = append(fds, fdSummary{ID: fdid, Lat: lat, Lon: lon})
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	494
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	495 if err := track(
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	496 ctx, tx, importID, "waterway.fairway_dimensions", fdid,
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	497 ); err != nil {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	498 return nil, err
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	499 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	500
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	501 old++
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	502 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	503
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	504 // Do not fail if features > 0 because otherwise new features are lost
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	505 if features == 0 && old == 0 {
5262 5df1eff5e445 Made staticcheck a little happier. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5144 diff changeset	506 return nil, fmt.Errorf("invalidating features failed")
5144 f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	507 }
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	508
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	509 if old > 0 {
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	510 feedback.Info("Number of features invalidated: %d", old)
f11b9b50fcc9 Keep historic data of fairway dimensions Tom Gottfried <tom@intevation.de> parents: 5035 diff changeset	511 }
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	512 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	513
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	514 if err = tx.Commit(); err == nil {
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	515 feedback.Info("Storing %d features took %s",
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	516 features, time.Since(start))
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	517 }
491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	518
1898 8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	519 summary := struct {
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	520 Date time.Time `json:"date"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	521 LOS int `json:"los"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	522 SourceOrganization string `json:"source-organization"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	523 FdArea []fdSummary `json:"fd-area"`
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	524 }{
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	525 Date: time.Now(),
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	526 LOS: fd.LOS,
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	527 SourceOrganization: fd.SourceOrganization,
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	528 FdArea: fds,
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	529 }
8a56a64e4410 Summarize the result of fairway dimension import for review. Raimund Renkert <raimund.renkert@intevation.de> parents: 1893 diff changeset	530 return &summary, err
1841 491f5b68da9e Implemented fairway dimension import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	531 }

Mercurial > gemma

annotate pkg/imports/fd.go @ 5560:f2204f91d286