Mercurial > gemma
annotate pkg/imports/wp.go @ 2723:a10022399e24
WFS downloads: Fetch user and password from config.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Tue, 19 Mar 2019 12:42:43 +0100 |
parents | 56c8ef515d93 |
children | 7cb027be277d |
rev | line source |
---|---|
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
17 "bufio" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
20 "encoding/csv" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
21 "encoding/json" |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
22 "errors" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
23 "fmt" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
24 "io" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "os" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
26 "path/filepath" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
27 "strconv" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
28 "strings" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
29 "time" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
31 "github.com/jackc/pgx/pgtype" |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
32 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
33 "gemma.intevation.de/gemma/pkg/misc" |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/models" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/wfs" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
38 // defaultPointToLinePrecision is the precision in meters |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
39 // to match from points to lines. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
40 const defaultPointToLinePrecision = 10 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
41 |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 type WaterwayProfiles struct { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 Dir string `json:"dir"` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
44 // URL the GetCapabilities URL of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
45 URL string `json:"url"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
46 // FeatureType selects the feature type of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
47 FeatureType string `json:"feature-type"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
48 // SortBy works around misconfigured services to |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
49 // establish a sort order to get the features. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
50 SortBy string `json:"sort-by"` |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
51 // Precsion of match points to line strings. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
52 Precision *float64 `json:"precision,omitempty"` |
2723
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
53 // User is an optional username for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
54 User string `json:"user,omitempty"` |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
55 // Password is an optional password for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
56 Password string `json:"password,omitempty"` |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 const WPJobKind JobKind = "wp" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 type wpJobCreator struct{} |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 func init() { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 RegisterJobCreator(WPJobKind, wpJobCreator{}) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2183
diff
changeset
|
67 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 func (wpJobCreator) AutoAccept() bool { return false } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 func (wpJobCreator) Description() string { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 return "waterway profiles" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 func (wpJobCreator) Depends() []string { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 return []string{ |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 "waterway_profiles", |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
81 const ( |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
82 createGeomTempTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
83 CREATE TEMP TABLE wp_geoms ( |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
84 geom geography(linestring, 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
85 ) ON COMMIT DROP` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
86 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
87 createTempIndexSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
88 CREATE INDEX ON wp_geoms USING GIST(geom)` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
89 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
90 analyzeTempTableSQL = `ANALYZE wp_geoms` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
91 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
92 insertGeomTmpTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
93 INSERT INTO wp_geoms (geom) VALUES ( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
94 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
95 )` |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
96 hasDistanceMarkSQL = ` |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
97 SELECT true FROM waterway.distance_marks_virtual |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
98 WHERE location_code = |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
99 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
100 LIMIT 1` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
101 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
102 insertWaterwayProfileSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
103 INSERT INTO waterway.waterway_profiles ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
104 location, |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
105 geom, |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
106 validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
107 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
108 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
109 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
110 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
111 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
112 date_info, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
113 source_organization |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
114 ) VALUES ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
115 ($1, $2, $3, $4, $5), |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
116 ( SELECT wp_geoms.geom |
2127
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
117 FROM wp_geoms, waterway.distance_marks_virtual AS dmv |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
118 WHERE dmv.location_code = |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
119 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
120 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
121 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true) |
2087
5d3d2e823314
Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2086
diff
changeset
|
122 LIMIT 1 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
123 ), |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
124 $6, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
125 $7, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
126 $8, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
127 $9, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
128 $10, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
129 $11, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
130 $12, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
131 $13 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
132 ) RETURNING id, geom is NULL` |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
133 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
134 wpStageDoneSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
135 UPDATE waterway.waterway_profiles SET staging_done = true |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
136 WHERE id IN ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
137 SELECT key FROM import.track_imports |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
138 WHERE import_id = $1 AND |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
139 relation = 'waterway.waterway_profiles'::regclass)` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
140 ) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
141 |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
142 func (wpJobCreator) StageDone( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
144 tx *sql.Tx, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
145 id int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
146 ) error { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
147 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
148 return err |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
149 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
150 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
151 func (wp *WaterwayProfiles) CleanUp() error { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
152 return os.RemoveAll(wp.Dir) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 func (wp *WaterwayProfiles) Do( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 importID int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 conn *sql.Conn, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 feedback Feedback, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 ) (interface{}, error) { |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
161 start := time.Now() |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
162 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
163 tx, err := conn.BeginTx(ctx, nil) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
164 if err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
165 return nil, err |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
166 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
167 defer tx.Rollback() |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
168 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
169 if err := wp.downloadGeometries( |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
170 ctx, importID, tx, start, feedback); err != nil { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
171 return nil, fmt.Errorf("error downloading geometries: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
172 } |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
173 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
174 summary, err := wp.processCSV( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
175 ctx, importID, tx, start, feedback) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
176 if err != nil { |
2143
f0641b5ad065
Waterway profile imports: Fixed typo in error message.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2127
diff
changeset
|
177 return nil, fmt.Errorf("error processing CSV: %v", err) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
178 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
179 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
180 if err := tx.Commit(); err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
181 return nil, fmt.Errorf( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
182 "Importing waterway profiles failed after %s: %v", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
183 time.Since(start), err) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
184 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
185 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
186 feedback.Info("Importing waterway profiles took %s", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
187 time.Since(start)) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
188 return summary, nil |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
189 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
190 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
191 func (wp *WaterwayProfiles) downloadGeometries( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
192 ctx context.Context, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
193 importID int64, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
194 tx *sql.Tx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
195 start time.Time, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
196 feedback Feedback, |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
197 ) error { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
198 feedback.Info("Start downloading geometries from WFS.") |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
199 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
200 feedback.Info("Loading capabilities from %s", wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
201 caps, err := wfs.GetCapabilities(wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
202 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
203 feedback.Error("Loading capabilities failed: %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
204 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
205 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
206 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
207 ft := caps.FindFeatureType(wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
208 if ft == nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
209 return fmt.Errorf("Unknown feature type '%s'", wp.FeatureType) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
210 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
211 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
212 feedback.Info("Found feature type '%s", wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
213 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
214 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
215 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
216 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
217 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
218 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
219 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
220 if wp.SortBy != "" { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
221 feedback.Info("Features will be sorted by '%s'", wp.SortBy) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
222 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
223 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
224 dl, err := wfs.GetFeatures(caps, wp.FeatureType, wp.SortBy) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
225 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
226 feedback.Error("Cannot create GetFeature URLs. %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
227 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
228 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
229 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
230 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
231 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
232 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
233 |
2126
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
234 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil { |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
235 return err |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
236 } |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
237 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
238 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
239 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
240 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
241 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
242 defer insertStmt.Close() |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
243 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
244 var ( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
245 unsupported = stringCounter{} |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
246 missingProperties int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
247 features int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
248 ) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
249 |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2622
diff
changeset
|
250 // TODO: Fetch from parameters. |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2622
diff
changeset
|
251 user, password := "", "" |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2622
diff
changeset
|
252 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
253 if err := dl.Download(user, password, func(url string, r io.Reader) error { |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2094
diff
changeset
|
254 feedback.Info("Get features from: '%s'", url) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
255 rfc, err := wfs.ParseRawFeatureCollection(r) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
256 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
257 return fmt.Errorf("parsing GetFeature document failed: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
258 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
259 if rfc.CRS != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
260 crsName := rfc.CRS.Properties.Name |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
261 if epsg, err = wfs.CRSToEPSG(crsName); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
262 feedback.Error("Unsupported CRS: %d", crsName) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
263 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
264 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
265 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
266 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
267 // No features -> ignore. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
268 if rfc.Features == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
269 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
270 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
271 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
272 for _, feature := range rfc.Features { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
273 if feature.Geometry.Coordinates == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
274 missingProperties++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
275 continue |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
276 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
277 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
278 switch feature.Geometry.Type { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
279 case "LineString": |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
280 var l lineSlice |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
281 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
282 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
283 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
284 if _, err := insertStmt.ExecContext( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
285 ctx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
286 l.asWKB(), |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
287 epsg, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
288 ); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
289 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
290 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
291 features++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
292 default: |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
293 unsupported[feature.Geometry.Type]++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
294 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
295 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
296 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
297 }); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
298 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
299 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
300 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
301 if missingProperties > 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
302 feedback.Warn("Missing properties: %d", missingProperties) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
303 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
304 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
305 if len(unsupported) != 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
306 feedback.Warn("Unsupported types found: %s", unsupported) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
307 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
308 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
309 if features == 0 { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
310 return errors.New("No features found") |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
311 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
312 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
313 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
314 } |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
315 return nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
316 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
317 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
318 func parseFloat64(s string) (sql.NullFloat64, error) { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
319 if s == "" { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
320 return sql.NullFloat64{}, nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
321 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
322 s = strings.Replace(s, ",", ".", -1) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
323 v, err := strconv.ParseFloat(s, 64) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
324 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
325 return sql.NullFloat64{}, err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
326 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
327 return sql.NullFloat64{Float64: v, Valid: true}, nil |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
328 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
329 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
330 func (wp *WaterwayProfiles) processCSV( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
331 ctx context.Context, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
332 importID int64, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
333 tx *sql.Tx, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
334 start time.Time, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
335 feedback Feedback, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
336 ) (interface{}, error) { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
337 feedback.Info("Start processing CSV file.") |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
338 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
339 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
340 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
341 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
342 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
343 defer f.Close() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
344 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
345 r := csv.NewReader(bufio.NewReader(f)) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
346 r.Comma = ';' |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
347 r.ReuseRecord = true |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
348 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
349 headers, err := r.Read() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
350 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
351 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
352 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
353 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
354 var ( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
355 locationIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
356 validFromIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
357 validToIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
358 lnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
359 mwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
360 hnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
361 fe30Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
362 fe100Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
363 dateInfoIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
364 sourceIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
365 ) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
366 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
367 fields := []struct { |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
368 idx *int |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
369 substr string |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
370 }{ |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
371 {&locationIdx, "location"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
372 {&validFromIdx, "valid_from"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
373 {&validToIdx, "valid_to"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
374 {&lnwlIdx, "lnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
375 {&mwlIdx, "mwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
376 {&hnwlIdx, "hnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
377 {&fe30Idx, "fe30"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
378 {&fe100Idx, "fe100"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
379 {&dateInfoIdx, "date_info"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
380 {&sourceIdx, "source"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
381 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
382 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
383 nextHeader: |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
384 for i, h := range headers { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
385 h = strings.ToLower(h) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
386 for j := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
387 if strings.Contains(h, fields[j].substr) { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
388 if *fields[j].idx != -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
389 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
390 "CSV has more than one column with name containing '%s'", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
391 fields[j].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
392 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
393 *fields[j].idx = i |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
394 continue nextHeader |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
395 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
396 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
397 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
398 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
399 var missing []string |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
400 for i := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
401 if *fields[i].idx == -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
402 missing = append(missing, fields[i].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
403 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
404 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
405 if len(missing) > 0 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
406 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
407 "CSV is missing columns: %s", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
408 strings.Join(missing, ", ")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
409 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
410 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
411 var precision float64 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
412 if wp.Precision != nil { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
413 if precision = *wp.Precision; precision < 0 { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
414 precision = -precision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
415 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
416 } else { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
417 precision = defaultPointToLinePrecision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
418 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
419 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
420 feedback.Info( |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
421 "Matching points to lines with a precision of %.4fm.", precision) |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
422 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
423 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
424 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
425 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
426 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
427 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
428 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
429 defer insertStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
430 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
431 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
432 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
433 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
434 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
435 defer trackStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
436 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
437 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
438 if err != nil { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
439 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
440 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
441 defer hasDistanceMarkStmt.Close() |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
442 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
443 var ids []int64 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
444 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
445 lines: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
446 for line := 1; ; line++ { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
447 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
448 row, err := r.Read() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
449 switch { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
450 case err == io.EOF || len(row) == 0: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
451 break lines |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
452 case err != nil: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
453 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
454 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
455 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
456 location, err := models.IsrsFromString(row[locationIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
457 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
458 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
459 "Invalid ISRS location code in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
460 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
461 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
462 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
463 var dummy bool |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
464 err = hasDistanceMarkStmt.QueryRowContext( |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
465 ctx, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
466 location.CountryCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
467 location.LoCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
468 location.FairwaySection, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
469 location.Orc, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
470 location.Hectometre, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
471 ).Scan(&dummy) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
472 |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
473 switch { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
474 case err == sql.ErrNoRows: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
475 feedback.Warn("No virtual distance mark found for %s.", location) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
476 continue lines |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
477 case err != nil: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
478 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
479 case !dummy: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
480 return nil, errors.New("unexpected result form database") |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
481 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
482 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
483 validFromTime, err := parseDate(row[validFromIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
484 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
485 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
486 "Invalid 'valid_from' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
487 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
488 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
489 validToTime, err := parseDate(row[validToIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
490 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
491 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
492 "Invalid 'valid_to' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
493 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
494 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
495 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
496 validFrom := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
497 Time: validFromTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
498 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
499 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
500 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
501 validTo := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
502 Time: validToTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
503 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
504 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
505 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
506 validity := pgtype.Tstzrange{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
507 Lower: validFrom, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
508 Upper: validTo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
509 LowerType: pgtype.Inclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
510 UpperType: pgtype.Exclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
511 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
512 } |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
513 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
514 lnwl, err := parseFloat64(row[lnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
515 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
516 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
517 "Invalid 'lnwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
518 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
519 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
520 mwl, err := parseFloat64(row[mwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
521 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
522 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
523 "Invalid 'mwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
524 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
525 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
526 hnwl, err := parseFloat64(row[hnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
527 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
528 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
529 "Invalid 'hnwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
530 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
531 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
532 fe30, err := parseFloat64(row[fe30Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
533 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
534 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
535 "Invalid 'fe30' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
536 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
537 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
538 fe100, err := parseFloat64(row[fe100Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
539 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
540 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
541 "Invalid 'fe100' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
542 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
543 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
544 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
545 var dateInfo time.Time |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
546 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
547 if di := row[dateInfoIdx]; di == "" { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
548 dateInfo = start |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
549 } else if dateInfo, err = parseDate(di); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
550 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
551 "Invalid 'date_info' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
552 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
553 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
554 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
555 source := row[sourceIdx] |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
556 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
557 var id int64 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
558 var noGeom bool |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
559 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
560 if err := insertStmt.QueryRowContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
561 ctx, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
562 location.CountryCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
563 location.LoCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
564 location.FairwaySection, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
565 location.Orc, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
566 location.Hectometre, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
567 &validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
568 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
569 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
570 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
571 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
572 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
573 dateInfo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
574 source, |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
575 precision, |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
576 ).Scan(&id, &noGeom); err != nil { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
577 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
578 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
579 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
580 if _, err := trackStmt.ExecContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
581 ctx, importID, "waterway.waterway_profiles", id); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
582 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
583 } |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
584 |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
585 if noGeom { |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
586 feedback.Warn( |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
587 "No profile geometry found for %s in line %d.", location, line) |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
588 } |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
589 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
590 ids = append(ids, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
591 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
592 if len(ids) == 0 { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
593 return nil, UnchangedError("No new entries in waterway profiles.") |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
594 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
595 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
596 feedback.Info("%d new entries in waterway profiles.", len(ids)) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
597 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
598 summary := struct { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
599 IDs []int64 `json:"ids"` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
600 }{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
601 IDs: ids, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
602 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
603 return &summary, nil |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
604 } |