Mercurial > gemma

annotate pkg/imports/wp.go @ 2126:d4a5e475f874

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Actually create index on temp table Makes import of waterway profiles roughly four times faster.
author Tom Gottfried <tom@intevation.de>
date Wed, 06 Feb 2019 14:27:14 +0100
parents 86c88fc0ff5e
children a1f2cfa624cf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
17 "bufio"
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "database/sql"
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
20 "encoding/csv"
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
22 "errors"
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
23 "fmt"
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
24 "io"
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "os"
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
26 "path/filepath"
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
27 "strconv"
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
28 "strings"
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
29 "time"
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30
2092
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
31 "github.com/jackc/pgx/pgtype"
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
32
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 "gemma.intevation.de/gemma/pkg/common"
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
34 "gemma.intevation.de/gemma/pkg/misc"
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
35 "gemma.intevation.de/gemma/pkg/models"
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
36 "gemma.intevation.de/gemma/pkg/wfs"
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37 )
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38
2094
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
39 // defaultPointToLinePrecision is the precision in meters
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
40 // to match from points to lines.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
41 const defaultPointToLinePrecision = 10
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
42
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 Dir string `json:"dir"`
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
45 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
46 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
47 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
48 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
49 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
50 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
51 SortBy string `json:"sort-by"`
2094
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
52 // Precsion of match points to line strings.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
53 Precision *float64 `json:"precision,omitempty"`
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 func (wpJobCreator) Create(_ JobKind, data string) (Job, error) {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 wp := new(WaterwayProfiles)
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 if err := common.FromJSONString(data, wp); err != nil {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 return nil, err
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 return wp, nil
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
74 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
75 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 func (wpJobCreator) Depends() []string {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79 return []string{
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80 "waterway_profiles",
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
82 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
83
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
84 const (
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
85 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
86 CREATE TEMP TABLE wp_geoms (
2093
a9d28c489761 Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2092
diff changeset
87 geom geography(linestring, 4326)
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
88 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
89
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
90 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
91 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
92
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
93 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
94
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
95 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
96 INSERT INTO wp_geoms (geom) VALUES (
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
97 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326)
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
98 )`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
99
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
100 insertWaterwayProfileSQL = `
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
101 WITH point AS (
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
102 SELECT
2094
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
103 ST_Buffer(geom, $14::float) AS geom,
2093
a9d28c489761 Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2092
diff changeset
104 geom AS point
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
105 FROM waterway.distance_marks_virtual
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
106 WHERE location_code =
2087
5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2086
diff changeset
107 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
108 )
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
109 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
110 location,
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
111 geom,
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
112 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
113 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
114 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
115 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
116 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
117 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
118 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
119 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
120 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
121 ($1, $2, $3, $4, $5),
2093
a9d28c489761 Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2092
diff changeset
122 ( SELECT wp_geoms.geom
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
123 FROM wp_geoms, point
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
124 WHERE wp_geoms.geom && point.geom
2093
a9d28c489761 Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2092
diff changeset
125 ORDER BY ST_Distance(point.point, wp_geoms.geom, true)
2087
5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2086
diff changeset
126 LIMIT 1
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
127 ),
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
128 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
129 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
130 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
131 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
132 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
133 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
134 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
135 $13
2092
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
136 ) RETURNING id, geom is NULL`
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
137
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
138 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
139 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
140 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
141 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
142 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
143 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
144 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
145
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
146 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
147 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
148 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
149 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
150 ) error {
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
151 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
152 return err
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
153 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
154
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
155 func (wp *WaterwayProfiles) CleanUp() error {
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
156 return os.RemoveAll(wp.Dir)
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157 }
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
158
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
160 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
161 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164 ) (interface{}, error) {
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
165 start := time.Now()
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
166
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
167 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
168 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
169 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
170 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
171 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
172
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
173 if err := wp.downloadGeometries(
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
174 ctx, importID, tx, start, feedback); err != nil {
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
175 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
176 }
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
177
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
178 summary, err := wp.processCSV(
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
179 ctx, importID, tx, start, feedback)
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
180 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
181 return nil, fmt.Errorf("error processing CVS: %v", err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
182 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
183
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
184 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
185 return nil, fmt.Errorf(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
186 "Importing waterway profiles failed after %s: %v",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
187 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
188 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
189
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
190 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
191 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
192 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
193 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
194
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
195 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
196 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
197 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
198 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
199 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
200 feedback Feedback,
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
201 ) error {
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
202 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
203
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
204 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
205 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
206 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
207 feedback.Error("Loading capabilities failed: %v", err)
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
208 return err
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
209 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
210
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
211 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
212 if ft == nil {
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
213 return fmt.Errorf("Unknown feature type '%s'", wp.FeatureType)
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
214 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
215
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
216 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
217
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
218 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
219 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
220 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
221 return err
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
222 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
223
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
224 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
225 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
226 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
227
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
228 urls, err := wfs.GetFeaturesGET(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
229 caps, wp.FeatureType, "application/json", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
230 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
231 feedback.Error("Cannot create GetFeature URLs. %v", err)
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
232 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
233 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
234
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
235 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
236 return err
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
237 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
238
2126
d4a5e475f874 Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents: 2109
diff changeset
239 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
d4a5e475f874 Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents: 2109
diff changeset
240 return err
d4a5e475f874 Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents: 2109
diff changeset
241 }
d4a5e475f874 Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents: 2109
diff changeset
242
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
243 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
244 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
245 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
246 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
247 defer insertStmt.Close()
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
248
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
249 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
250 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
251 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
252 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
253 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
254
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2094
diff changeset
255 if err := wfs.DownloadURLs(urls, func(url string, r io.Reader) error {
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2094
diff changeset
256 feedback.Info("Get features from: '%s'", url)
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
257 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
258 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
259 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
260 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
261 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
262 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
263 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
264 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
265 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
266 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
267 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
268
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
269 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
270 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
271 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
272 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
273
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
274 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
275 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
276 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
277 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
278 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
279
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
280 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
281 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
282 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
283 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
284 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
285 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
286 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
287 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
288 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
289 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
290 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
291 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
292 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
293 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
294 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
295 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
296 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
297 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
298 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
299 }); err != nil {
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
300 return err
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
301 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
302
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
303 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
304 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
305 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
306
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
307 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
308 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
309 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
310
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
311 if features == 0 {
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
312 return errors.New("No features found")
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
313 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
314 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
315 return err
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
316 }
2091
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
317 return nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
318 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
319
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
320 func parseFloat64(s string) (sql.NullFloat64, error) {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
321 if s == "" {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
322 return sql.NullFloat64{}, nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
323 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
324 s = strings.Replace(s, ",", ".", -1)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
325 v, err := strconv.ParseFloat(s, 64)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
326 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
327 return sql.NullFloat64{}, err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
328 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2087
diff changeset
329 return sql.NullFloat64{Float64: v, Valid: true}, nil
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
330 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
331
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
332 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
333 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
334 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
335 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
336 start time.Time,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
337 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
338 ) (interface{}, error) {
2086
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2084
diff changeset
339 feedback.Info("Start processing CSV file.")
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
340
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
341 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
342 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
343 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
344 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
345 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
346
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
347 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
348 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
349 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
350
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
351 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
352 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
353 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
354 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
355
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
356 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
357 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
358 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
359 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
360 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
361 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
362 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
363 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
364 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
365 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
366 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
367 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
368
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
369 fields := []struct {
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
370 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
371 substr string
2084
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2079
diff changeset
372 }{
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
373 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
374 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
375 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
376 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
377 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
378 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
379 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
380 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
381 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
382 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
383 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
384
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
385 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
386 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
387 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
388 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
389 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
390 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
391 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
392 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
393 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
394 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
395 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
396 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
397 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
398 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
399 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
400
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
401 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
402 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
403 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
404 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
405 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
406 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
407 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
408 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
409 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
410 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
411 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
412
2094
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
413 var precision float64
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
414 if wp.Precision != nil {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
415 if precision = *wp.Precision; precision < 0 {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
416 precision = -precision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
417 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
418 } else {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
419 precision = defaultPointToLinePrecision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
420 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
421
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
422 feedback.Info(
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
423 "Matching points to lines with a precision of %.4fm.", precision)
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
424
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
425 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
426
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
427 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
428 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
429 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
430 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
431 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
432
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
433 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
434 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
435 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
436 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
437 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
438
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
439 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
440
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
441 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
442 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
443
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
444 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
445 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
446 case err == io.EOF || len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
447 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
448 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
449 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
450 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
451
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
452 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
453 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
454 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
455 "Invalid ISRS location code in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
456 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
457 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
458
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
459 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
460 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
461 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
462 "Invalid 'valid_from' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
463 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
464 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
465 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
466 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
467 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
468 "Invalid 'valid_to' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
469 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
470 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
471
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
472 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
473 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
474 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
475 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
476
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
477 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
478 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
479 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
480 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
481
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
482 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
483 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
484 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
485 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
486 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
487 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
488 }
2078
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2072
diff changeset
489
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
490 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
491 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
492 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
493 "Invalid 'lnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
494 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
495 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
496 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
497 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
498 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
499 "Invalid 'mwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
500 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
501 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
502 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
503 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
504 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
505 "Invalid 'hnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
506 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
507 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
508 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
509 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
510 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
511 "Invalid 'fe30' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
512 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
513 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
514 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
515 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
516 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
517 "Invalid 'fe100' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
518 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
519 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
520
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
521 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
522
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
523 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
524 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
525 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
526 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
527 "Invalid 'date_info' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
528 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
529 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
530
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
531 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
532
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
533 var id int64
2092
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
534 var noGeom bool
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
535
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
536 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
537 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
538 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
539 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
540 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
541 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
542 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
543 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
544 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
545 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
546 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
547 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
548 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
549 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
550 source,
2094
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2093
diff changeset
551 precision,
2092
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
552 ).Scan(&id, &noGeom); err != nil {
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
553 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
554 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
555
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
556 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
557 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
558 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
559 }
2092
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
560
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
561 if noGeom {
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
562 feedback.Warn(
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
563 "No profile geometry found for %s in line %d.", location, line)
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
564 }
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2091
diff changeset
565
2079
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
566 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
567 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
568 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
569 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
570 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
571
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
572 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
573
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
574 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
575 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
576 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
577 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
578 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2078
diff changeset
579 return &summary, nil
2072
b4d8d320feab Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
580 }