Mercurial > gemma

annotate pkg/imports/wa.go @ 2721:56c8ef515d93

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
WFS downloader: Use new GML/GeoJSON factory function in gemma.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Tue, 19 Mar 2019 12:28:22 +0100
parents 2b6f73c30016
children a10022399e24
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
3 //
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
6 //
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
10 //
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
14
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
15 package imports
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
16
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
17 import (
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
18 "context"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
19 "database/sql"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
20 "encoding/json"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
21 "errors"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
22 "fmt"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
23 "io"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
24 "strconv"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
25 "time"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
26
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
27 "gemma.intevation.de/gemma/pkg/wfs"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
28 )
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
29
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
30 // WaterwayArea is an import job to import
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
31 // the waterway area in form of polygon geometries
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
32 // and attribute data from a WFS service.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
33 type WaterwayArea struct {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
34 // URL the GetCapabilities URL of the WFS service.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
35 URL string `json:"url"`
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
36 // FeatureType selects the feature type of the WFS service.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
37 FeatureType string `json:"feature-type"`
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
38 // SortBy works around misconfigured services to
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
39 // establish a sort order to get the features.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
40 SortBy string `json:"sort-by"`
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
41 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
42
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
43 // WAJobKind is the import queue type identifier.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
44 const WAJobKind JobKind = "wa"
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
45
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
46 type waJobCreator struct{}
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
47
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
48 func init() {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
49 RegisterJobCreator(WAJobKind, waJobCreator{})
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
50 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
51
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
52 func (waJobCreator) Description() string { return "waterway area" }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
53
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
54 func (waJobCreator) AutoAccept() bool { return true }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
55
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
56 func (waJobCreator) Create() Job { return new(WaterwayArea) }
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
57
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
58 func (waJobCreator) Depends() []string {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
59 return []string{
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
60 "waterway_area",
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
61 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
62 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
63
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
64 // StageDone is a NOP for waterway area imports.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
65 func (waJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
66 return nil
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
67 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
68
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
69 // CleanUp for waterway area imports is a NOP.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
70 func (*WaterwayArea) CleanUp() error { return nil }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
71
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
72 type waterwayAreaProperties struct {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
73 Catccl *string `json:"ienc_catccl"`
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
74 Dirimp *string `json:"ienc_dirimp"`
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
75 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
76
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
77 const (
1796
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
78 deleteWaterwayAreaSQL = `
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
79 WITH resp AS (
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
80 SELECT best_utm(area::geometry) AS t,
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
81 ST_Transform(area::geometry, best_utm(area::geometry)) AS a
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
82 FROM users.responsibility_areas
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
83 WHERE country = users.current_user_country()
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
84 )
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
85 DELETE FROM waterway.waterway_area
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
86 WHERE pg_has_role('sys_admin', 'MEMBER')
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
87 OR ST_Covers((SELECT a FROM resp),
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
88 ST_Transform(area::geometry, (SELECT t FROM resp)))
1796
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
89 `
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
90 insertWaterwayAreaSQL = `
1796
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
91 WITH resp AS (
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
92 SELECT best_utm(area::geometry) AS t,
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
93 ST_Transform(area::geometry, best_utm(area::geometry)) AS a
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
94 FROM users.responsibility_areas
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
95 WHERE country = users.current_user_country()
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
96 )
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
97 INSERT INTO waterway.waterway_area (area, catccl, dirimp)
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
98 SELECT ST_Transform(clipped.geom, 4326)::geography, $3, $4
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
99 FROM resp,
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
100 ST_CollectionExtract(ST_MakeValid(ST_Transform(
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
101 ST_GeomFromWKB($1, $2::integer), t)), 3) AS new_area (new_area),
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
102 LATERAL (SELECT (ST_Dump(
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
103 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
104 THEN new_area
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
105 ELSE ST_Intersection(a, new_area)
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
106 END
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
107 )).geom AS geom
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
108 ) AS clipped
1796
904c7e853adc Waterway area import: Clip features against responsibility area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1786
diff changeset
109 `
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
110 )
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
111
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
112 // Do executes the actual waterway axis import.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
113 func (wx *WaterwayArea) Do(
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
114 ctx context.Context,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
115 importID int64,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
116 conn *sql.Conn,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
117 feedback Feedback,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
118 ) (interface{}, error) {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
119
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
120 start := time.Now()
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
121
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
122 feedback.Info("Import waterway area")
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
123
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
124 feedback.Info("Loading capabilities from %s", wx.URL)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
125 caps, err := wfs.GetCapabilities(wx.URL)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
126 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
127 feedback.Error("Loading capabilities failed: %v", err)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
128 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
129 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
130
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
131 ft := caps.FindFeatureType(wx.FeatureType)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
132 if ft == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
133 return nil, fmt.Errorf("Unknown feature type '%s'", wx.FeatureType)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
134 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
135
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
136 feedback.Info("Found feature type '%s", wx.FeatureType)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
137
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
138 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
139 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
140 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
141 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
142 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
143
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
144 if wx.SortBy != "" {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
145 feedback.Info("Features will be sorted by '%s'", wx.SortBy)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
146 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
147
2721
56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
148 dl, err := wfs.GetFeatures(caps, wx.FeatureType, wx.SortBy)
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
149 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
150 feedback.Error("Cannot create GetFeature URLs. %v", err)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
151 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
152 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
153
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
154 tx, err := conn.BeginTx(ctx, nil)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
155 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
156 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
157 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
158 defer tx.Rollback()
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
159
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
160 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAreaSQL)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
161 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
162 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
163 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
164 defer insertStmt.Close()
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
165
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
166 // Delete the old features.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
167 if _, err := tx.ExecContext(ctx, deleteWaterwayAreaSQL); err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
168 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
169 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
170
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
171 var (
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
172 unsupported = stringCounter{}
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
173 missingProperties int
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
174 badProperties int
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
175 features int
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
176 )
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
177
2719
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2661
diff changeset
178 // TODO: Fetch from parameters.
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2661
diff changeset
179 user, password := "", ""
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2661
diff changeset
180
2721
56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
181 if err := dl.Download(user, password, func(url string, r io.Reader) error {
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1899
diff changeset
182 feedback.Info("Get features from: '%s'", url)
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
183 rfc, err := wfs.ParseRawFeatureCollection(r)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
184 if err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
185 return fmt.Errorf("parsing GetFeature document failed: %v", err)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
186 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
187 if rfc.CRS != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
188 crsName := rfc.CRS.Properties.Name
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
189 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
190 feedback.Error("Unsupported CRS: %d", crsName)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
191 return err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
192 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
193 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
194
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
195 // No features -> ignore.
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
196 if rfc.Features == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
197 return nil
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
198 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
199
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
200 feedback.Info("Using EPSG: %d", epsg)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
201
2659
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
202 const featureSavepoint = "feature"
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
203
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
204 for _, feature := range rfc.Features {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
205 if feature.Properties == nil || feature.Geometry.Coordinates == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
206 missingProperties++
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
207 continue
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
208 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
209
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
210 var props waterwayAreaProperties
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
211
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
212 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
213 badProperties++
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
214 continue
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
215 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
216
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
217 var catccl sql.NullInt64
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
218 if props.Catccl != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
219 if value, err := strconv.ParseInt(*props.Catccl, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
220 catccl = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
221 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
222 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
223 var dirimp sql.NullInt64
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
224 if props.Dirimp != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
225 if value, err := strconv.ParseInt(*props.Dirimp, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
226 dirimp = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
227 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
228 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
229
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
230 switch feature.Geometry.Type {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
231 case "Polygon":
1786
09349ca27dd7 Imports: Removed duplicated code path to store WKB polygons into database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1785
diff changeset
232 var p polygonSlice
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
233 if err := json.Unmarshal(*feature.Geometry.Coordinates, &p); err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
234 return err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
235 }
2661
542d3441c2d8 Bring context.Context parameters to front.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2659
diff changeset
236 if err := Savepoint(ctx, tx, featureSavepoint); err != nil {
2659
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
237 return err
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
238 }
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
239 if _, err := insertStmt.ExecContext(
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
240 ctx,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
241 p.asWKB(),
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
242 epsg,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
243 catccl,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
244 dirimp,
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
245 ); err != nil {
2659
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
246 feedback.Warn(handleError(err).Error())
2661
542d3441c2d8 Bring context.Context parameters to front.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2659
diff changeset
247 if err = RollbackToSavepoint(ctx, tx, featureSavepoint); err != nil {
2659
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
248 return err
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
249 }
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
250 } else {
2661
542d3441c2d8 Bring context.Context parameters to front.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2659
diff changeset
251 if err = ReleaseSavepoint(ctx, tx, featureSavepoint); err != nil {
2659
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
252 return err
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
253 }
00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails
Tom Gottfried <tom@intevation.de>
parents: 2649
diff changeset
254 features++
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
255 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
256 default:
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
257 unsupported[feature.Geometry.Type]++
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
258 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
259 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
260 return nil
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
261 }); err != nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
262 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
263 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
264
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
265 if badProperties > 0 {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
266 feedback.Warn("Bad properties: %d", badProperties)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
267 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
268
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
269 if missingProperties > 0 {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
270 feedback.Warn("Missing properties: %d", missingProperties)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
271 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
272
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
273 if len(unsupported) != 0 {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
274 feedback.Warn("Unsupported types found: %s", unsupported)
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
275 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
276
1828
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
277 if features == 0 {
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
278 err := errors.New("No features found")
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
279 feedback.Error("%v", err)
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
280 return nil, err
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
281 }
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1796
diff changeset
282
1785
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
283 if err = tx.Commit(); err == nil {
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
284 feedback.Info("Storing %d features took %s",
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
285 features, time.Since(start))
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
286 }
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
287
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
288 return nil, err
614c6c766691 Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
289 }