Mercurial > gemma

annotate pkg/imports/dma.go @ 2006:35acb7f9ae0c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Do anything else before expectedly failing role creation Creating roles during database setup expectedly fails in case there already is another gemma database in the cluster. Doing it at the end of the transaction ensures it does not hide errors in other commands in the script. In passing, add the default admin via the designated view to ensure it will become a correctly set up application user.
author Tom Gottfried <tom@intevation.de>
date Thu, 24 Jan 2019 17:23:43 +0100
parents 427f86518097
children 2b72f5e005aa
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1862
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
3 //
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
6 //
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
10 //
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
11 // Author(s):
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
12 // * Raimund Renkert <raimund.renkert@intevation.de>
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
13
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
14 package imports
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
15
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
16 import (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
17 "context"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
18 "database/sql"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
19 "encoding/json"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
20 "errors"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
21 "fmt"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
22 "io"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
23 "time"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
24
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/common"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
26 "gemma.intevation.de/gemma/pkg/wfs"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
27 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
28
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
29 // FairwayDimension is an import job to import
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
30 // the fairway dimensions in form of polygon geometries
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
31 // and attribute data from a WFS service.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
32 type DistanceMarksAshore struct {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
33 // URL the GetCapabilities URL of the WFS service.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
34 URL string `json:"url"`
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
35 // FeatureType selects the feature type of the WFS service.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
36 FeatureType string `json:"feature-type"`
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
37 // SortBy sorts the feature by this key.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
38 SortBy string `json:"sort-by"`
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
39 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
40
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
41 // DMAJobKind is the import queue type identifier.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
42 const DMAJobKind JobKind = "dma"
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
43
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
44 type dmaJobCreator struct{}
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
45
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
46 func init() {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
47 RegisterJobCreator(DMAJobKind, dmaJobCreator{})
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
48 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
49
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
50 func (dmaJobCreator) Description() string { return "distance marks" }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
51
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
52 func (dmaJobCreator) AutoAccept() bool { return true }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
53
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
54 func (dmaJobCreator) Create(_ JobKind, data string) (Job, error) {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
55 dma := new(DistanceMarksAshore)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
56 if err := common.FromJSONString(data, dma); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
57 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
58 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
59 return dma, nil
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
60 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
61
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
62 func (dmaJobCreator) Depends() []string {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
63 return []string{
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
64 "distance_marks",
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
65 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
66 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
67
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
68 // StageDone is a NOP for distance marks imports.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
69 func (dmaJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
70 return nil
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
71 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
72
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
73 // CleanUp for distance marks imports is a NOP.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
74 func (*DistanceMarksAshore) CleanUp() error { return nil }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
75
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
76 type distanceMarksAshoreProperties struct {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
77 HydroCatdis int `json:"hydro_catdis"`
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
78 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
79
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
80 const (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
81 deleteDistanceMarksSQL = `
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
82 WITH resp AS (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
83 SELECT best_utm(area::geometry) AS t,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
84 ST_Transform(area::geometry, best_utm(area::geometry)) AS a
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
85 FROM users.responsibility_areas
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
86 WHERE country = users.current_user_country()
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
87 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
88 DELETE FROM waterway.distance_marks
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
89 WHERE ST_Covers(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
90 (SELECT a FROM resp),
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
91 ST_Transform(geom::geometry, (SELECT t FROM resp)))
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
92 `
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
93 insertDistanceMarksSQL = `
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
94 WITH resp AS (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
95 SELECT best_utm(area::geometry) AS t,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
96 ST_Transform(area::geometry, best_utm(area::geometry)) AS a
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
97 FROM users.responsibility_areas
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
98 WHERE country = users.current_user_country()
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
99 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
100 INSERT INTO waterway.distance_marks (geom, catdis)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
101 SELECT ST_Transform(clipped.geom, 4326)::geography, $3 FROM (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
102 SELECT (ST_Dump(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
103 ST_Intersection(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
104 (SELECT a FROM resp),
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
105 ST_Transform(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
106 ST_GeomFromWKB($1, $2::integer),
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
107 (SELECT t FROM resp)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
108 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
109 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
110 )).geom AS geom
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
111 ) AS clipped
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
112 WHERE clipped.geom IS NOT NULL
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
113 `
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
114 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
115
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
116 // Do executes the actual fairway dimension import.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
117 func (dma *DistanceMarksAshore) Do(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
118 ctx context.Context,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
119 importID int64,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
120 conn *sql.Conn,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
121 feedback Feedback,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
122 ) (interface{}, error) {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
123
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
124 start := time.Now()
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
125
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
126 feedback.Info("Import distance marks")
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
127
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
128 feedback.Info("Loading capabilities from %s", dma.URL)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
129 caps, err := wfs.GetCapabilities(dma.URL)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
130 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
131 feedback.Error("Loading capabilities failed: %v", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
132 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
133 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
134
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
135 ft := caps.FindFeatureType(dma.FeatureType)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
136 if ft == nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
137 return nil, fmt.Errorf("Unknown feature type '%s'", dma.FeatureType)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
138 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
139
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
140 feedback.Info("Found feature type '%s'", dma.FeatureType)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
141
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
142 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
143 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
144 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
145 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
146 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
147
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
148 urls, err := wfs.GetFeaturesGET(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
149 caps, dma.FeatureType, "application/json", dma.SortBy)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
150 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
151 feedback.Error("Cannot create GetFeature URLs. %v", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
152 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
153 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
154
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
155 tx, err := conn.BeginTx(ctx, nil)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
156 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
157 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
158 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
159 defer tx.Rollback()
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
160
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
161 insertStmt, err := tx.PrepareContext(ctx, insertDistanceMarksSQL)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
162 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
163 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
164 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
165 defer insertStmt.Close()
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
166
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
167 // Delete the old features.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
168 if _, err := tx.ExecContext(ctx, deleteDistanceMarksSQL); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
169 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
170 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
171
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
172 var (
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
173 unsupported = stringCounter{}
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
174 missingProperties int
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
175 badProperties int
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
176 features int
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
177 )
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
178
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
179 if err := wfs.DownloadURLs(urls, func(r io.Reader) error {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
180 rfc, err := wfs.ParseRawFeatureCollection(r)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
181 if err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
182 return fmt.Errorf("parsing GetFeature document failed: %v", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
183 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
184 if rfc.CRS != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
185 crsName := rfc.CRS.Properties.Name
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
186 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
187 feedback.Error("Unsupported CRS: %d", crsName)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
188 return err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
189 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
190 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
191
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
192 // No features -> ignore.
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
193 if rfc.Features == nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
194 return nil
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
195 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
196
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
197 feedback.Info("Using EPSG: %d", epsg)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
198
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
199 for _, feature := range rfc.Features {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
200 if feature.Geometry.Coordinates == nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
201 missingProperties++
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
202 continue
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
203 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
204
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
205 var props distanceMarksAshoreProperties
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
206
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
207 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
208 badProperties++
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
209 continue
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
210 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
211 switch feature.Geometry.Type {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
212 case "Point":
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
213 var p pointSlice
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
214 if err := json.Unmarshal(*feature.Geometry.Coordinates, &p); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
215 return err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
216 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
217 if _, err := insertStmt.ExecContext(
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
218 ctx,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
219 p.asWKB(),
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
220 epsg,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
221 props.HydroCatdis,
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
222 ); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
223 feedback.Error("error: %s", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
224 return err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
225 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
226 features++
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
227 default:
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
228 unsupported[feature.Geometry.Type]++
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
229 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
230 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
231 return nil
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
232 }); err != nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
233 feedback.Error("Downloading features failed: %v", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
234 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
235 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
236
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
237 if badProperties > 0 {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
238 feedback.Warn("Bad properties: %d", badProperties)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
239 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
240
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
241 if missingProperties > 0 {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
242 feedback.Warn("Missing properties: %d", missingProperties)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
243 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
244
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
245 if len(unsupported) != 0 {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
246 feedback.Warn("Unsupported types found: %s", unsupported)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
247 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
248
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
249 if features == 0 {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
250 err := errors.New("No features found")
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
251 feedback.Error("%v", err)
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
252 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
253 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
254
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
255 if err = tx.Commit(); err == nil {
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
256 feedback.Info("Storing %d features took %s",
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
257 features, time.Since(start))
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
258 }
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
259
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
260 return nil, err
427f86518097 Added distance marks importer in backend.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
261 }