gemma: pkg/imports/wa.go annotate

annotate pkg/imports/wa.go @ 2855:8a8a929182f9

Enable import of axis and area independently from areas of responsibility In preparation of generating the areas of responsibility from data previously imported by a sys_admin! Having the CTE as an item in the FROM-list prevents the SELECT from returning any rows if the CTE returns no rows. Thus, use the CTE only in scalar subqueries that are relevant only if the statement is executed by somebody without sys_admin role. Also introduced an ST_CollectionExtract() in the axis INSERT because here too, the intersection is not guaranteed to have the right dimensionality.

author	Tom Gottfried <tom@intevation.de>
date	Thu, 28 Mar 2019 17:19:27 +0100
parents	1b6840093eac
children	93fa55bce126

rev	line source
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	3 //
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	6 //
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	10 //
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	11 // Author(s):
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	14
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	15 package imports
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	16
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	17 import (
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	18 "context"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	19 "database/sql"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	20 "encoding/json"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	21 "errors"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	22 "fmt"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	23 "io"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	24 "strconv"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	25 "time"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	26
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	27 "gemma.intevation.de/gemma/pkg/wfs"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	28 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	29
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	30 // WaterwayArea is an import job to import
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	31 // the waterway area in form of polygon geometries
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	32 // and attribute data from a WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	33 type WaterwayArea struct {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	34 // URL the GetCapabilities URL of the WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	35 URL string `json:"url"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	36 // FeatureType selects the feature type of the WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	37 FeatureType string `json:"feature-type"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	38 // SortBy works around misconfigured services to
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	39 // establish a sort order to get the features.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	40 SortBy string `json:"sort-by"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	41 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	42 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	43 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	44 Password string `json:"password,omitempty"`
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	45 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	46
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	47 // WAJobKind is the import queue type identifier.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	48 const WAJobKind JobKind = "wa"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	49
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	50 type waJobCreator struct{}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	51
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	52 func init() {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	53 RegisterJobCreator(WAJobKind, waJobCreator{})
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	54 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	55
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	56 func (waJobCreator) Description() string { return "waterway area" }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	57
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	58 func (waJobCreator) AutoAccept() bool { return true }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	59
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	60 func (waJobCreator) Create() Job { return new(WaterwayArea) }
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	61
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	62 func (waJobCreator) Depends() []string {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	63 return []string{
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	64 "waterway_area",
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	65 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	66 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	67
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	68 // StageDone is a NOP for waterway area imports.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	69 func (waJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	70 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	71 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	72
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	73 // CleanUp for waterway area imports is a NOP.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	74 func (*WaterwayArea) CleanUp() error { return nil }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	75
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	76 type waterwayAreaProperties struct {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	77 Catccl *string `json:"ienc_catccl"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	78 Dirimp *string `json:"ienc_dirimp"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	79 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	80
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	81 const (
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	82 deleteWaterwayAreaSQL = `
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	83 WITH resp AS (
2833 1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2758 diff changeset	84 SELECT best_utm(area) AS t,
1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2758 diff changeset	85 ST_Transform(area::geometry, best_utm(area)) AS a
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	86 FROM users.responsibility_areas
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	87 WHERE country = users.current_user_country()
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	88 )
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	89 DELETE FROM waterway.waterway_area
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	90 WHERE pg_has_role('sys_admin', 'MEMBER')
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	91 OR ST_Covers((SELECT a FROM resp),
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	92 ST_Transform(area::geometry, (SELECT t FROM resp)))
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	93 `
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	94 insertWaterwayAreaSQL = `
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	95 WITH resp AS (
2833 1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2758 diff changeset	96 SELECT best_utm(area) AS t,
1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2758 diff changeset	97 ST_Transform(area::geometry, best_utm(area)) AS a
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	98 FROM users.responsibility_areas
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	99 WHERE country = users.current_user_country()
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	100 )
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	101 INSERT INTO waterway.waterway_area (area, catccl, dirimp)
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	102 SELECT dmp.geom, $3, $4
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	103 FROM ST_GeomFromWKB($1, $2::integer) AS new_area (new_area),
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	104 ST_Dump(ST_Transform(ST_CollectionExtract(
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	105 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	106 THEN ST_MakeValid(ST_Transform(new_area,
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	107 best_utm(ST_Transform(new_area, 4326))))
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	108 ELSE ST_Intersection((SELECT a FROM resp),
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	109 ST_MakeValid(ST_Transform(new_area, (SELECT t FROM resp))))
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	110 END,
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	111 3), 4326)) AS dmp
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	112 `
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	113 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	114
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	115 // Do executes the actual waterway axis import.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	116 func (wx *WaterwayArea) Do(
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	117 ctx context.Context,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	118 importID int64,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	119 conn *sql.Conn,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	120 feedback Feedback,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	121 ) (interface{}, error) {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	122
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	123 start := time.Now()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	124
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	125 feedback.Info("Import waterway area")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	126
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	127 feedback.Info("Loading capabilities from %s", wx.URL)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	128 caps, err := wfs.GetCapabilities(wx.URL)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	129 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	130 feedback.Error("Loading capabilities failed: %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	131 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	132 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	133
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	134 ft := caps.FindFeatureType(wx.FeatureType)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	135 if ft == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	136 return nil, fmt.Errorf("Unknown feature type '%s'", wx.FeatureType)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	137 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	138
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	139 feedback.Info("Found feature type '%s", wx.FeatureType)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	140
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	141 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	142 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	143 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	144 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	145 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	146
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	147 if wx.SortBy != "" {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	148 feedback.Info("Features will be sorted by '%s'", wx.SortBy)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	149 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	150
2721 56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2719 diff changeset	151 dl, err := wfs.GetFeatures(caps, wx.FeatureType, wx.SortBy)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	152 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	153 feedback.Error("Cannot create GetFeature URLs. %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	154 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	155 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	156
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	157 tx, err := conn.BeginTx(ctx, nil)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	158 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	159 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	160 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	161 defer tx.Rollback()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	162
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	163 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAreaSQL)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	164 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	165 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	166 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	167 defer insertStmt.Close()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	168
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	169 // Delete the old features.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	170 if _, err := tx.ExecContext(ctx, deleteWaterwayAreaSQL); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	171 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	172 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	173
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	174 var (
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	175 unsupported = stringCounter{}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	176 missingProperties int
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	177 badProperties int
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	178 features int
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	179 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	180
2729 7cb027be277d WFS downloader: Really use username and password from parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2723 diff changeset	181 if err := dl.Download(wx.User, wx.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1899 diff changeset	182 feedback.Info("Get features from: '%s'", url)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	183 rfc, err := wfs.ParseRawFeatureCollection(r)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	184 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	185 return fmt.Errorf("parsing GetFeature document failed: %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	186 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	187 if rfc.CRS != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	188 crsName := rfc.CRS.Properties.Name
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	189 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	190 feedback.Error("Unsupported CRS: %d", crsName)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	191 return err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	192 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	193 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	194
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	195 // No features -> ignore.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	196 if rfc.Features == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	197 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	198 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	199
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	200 feedback.Info("Using EPSG: %d", epsg)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	201
2758 a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	202 savepoint := Savepoint(ctx, tx, "feature")
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	203
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	204 for _, feature := range rfc.Features {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	205 if feature.Properties == nil \|\| feature.Geometry.Coordinates == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	206 missingProperties++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	207 continue
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	208 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	209
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	210 var props waterwayAreaProperties
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	211
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	212 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	213 badProperties++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	214 continue
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	215 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	216
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	217 var catccl sql.NullInt64
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	218 if props.Catccl != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	219 if value, err := strconv.ParseInt(*props.Catccl, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	220 catccl = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	221 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	222 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	223 var dirimp sql.NullInt64
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	224 if props.Dirimp != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	225 if value, err := strconv.ParseInt(*props.Dirimp, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	226 dirimp = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	227 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	228 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	229
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	230 switch feature.Geometry.Type {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	231 case "Polygon":
1786 09349ca27dd7 Imports: Removed duplicated code path to store WKB polygons into database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	232 var p polygonSlice
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	233 if err := json.Unmarshal(*feature.Geometry.Coordinates, &p); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	234 return err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	235 }
2758 a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	236 if err := savepoint(func() error {
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	237 _, err := insertStmt.ExecContext(
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	238 ctx,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	239 p.asWKB(),
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	240 epsg,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	241 catccl,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	242 dirimp,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	243 )
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	244 return err
2758 a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	245 }); err != nil {
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	246 feedback.Warn(handleError(err).Error())
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	247 } else {
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	248 features++
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	249 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	250 default:
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	251 unsupported[feature.Geometry.Type]++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	252 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	253 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	254 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	255 }); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	256 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	257 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	258
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	259 if badProperties > 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	260 feedback.Warn("Bad properties: %d", badProperties)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	261 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	262
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	263 if missingProperties > 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	264 feedback.Warn("Missing properties: %d", missingProperties)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	265 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	266
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	267 if len(unsupported) != 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	268 feedback.Warn("Unsupported types found: %s", unsupported)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	269 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	270
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	271 if features == 0 {
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	272 err := errors.New("No features found")
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	273 feedback.Error("%v", err)
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	274 return nil, err
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	275 }
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	276
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	277 if err = tx.Commit(); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	278 feedback.Info("Storing %d features took %s",
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	279 features, time.Since(start))
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	280 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	281
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	282 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	283 }

Mercurial > gemma

annotate pkg/imports/wa.go @ 2855:8a8a929182f9