Mercurial > gemma
annotate pkg/imports/wp.go @ 3705:7006b92c0334
Handle updates (vs. historized and new versions) separately.
We need this distinction as updated data currently can not be
reviewed. More precisely: it can not be declined after review, as the
old data is updated in place.
The current exclusion from the review is a workaround and not meant to
be the final solution. Note that there are additional minor problems,
like the fact that the updated data is not counted as changed data for
the import.
author | Sascha Wilde <wilde@intevation.de> |
---|---|
date | Wed, 19 Jun 2019 17:00:08 +0200 |
parents | 4acbee65275d |
children | eb2f949ddfa2 |
rev | line source |
---|---|
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
17 "bufio" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
20 "encoding/csv" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
21 "encoding/json" |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
22 "errors" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
23 "fmt" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
24 "io" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "os" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
26 "path/filepath" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
27 "strconv" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
28 "strings" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
29 "time" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
31 "github.com/jackc/pgx/pgtype" |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
32 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
33 "gemma.intevation.de/gemma/pkg/misc" |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/models" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/wfs" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
38 // defaultPointToLinePrecision is the precision in meters |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
39 // to match from points to lines. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
40 const defaultPointToLinePrecision = 10 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
41 |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 type WaterwayProfiles struct { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 Dir string `json:"dir"` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
44 // URL the GetCapabilities URL of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
45 URL string `json:"url"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
46 // FeatureType selects the feature type of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
47 FeatureType string `json:"feature-type"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
48 // SortBy works around misconfigured services to |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
49 // establish a sort order to get the features. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
50 SortBy string `json:"sort-by"` |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
51 // Precsion of match points to line strings. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
52 Precision *float64 `json:"precision,omitempty"` |
2723
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
53 // User is an optional username for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
54 User string `json:"user,omitempty"` |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
55 // Password is an optional password for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
56 Password string `json:"password,omitempty"` |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 const WPJobKind JobKind = "wp" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 type wpJobCreator struct{} |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 func init() { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 RegisterJobCreator(WPJobKind, wpJobCreator{}) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2183
diff
changeset
|
67 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 func (wpJobCreator) AutoAccept() bool { return false } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 func (wpJobCreator) Description() string { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 return "waterway profiles" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
75 func (wpJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
76 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
77 {"waterway_profiles"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
78 {"distance_marks_virtual"}, |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
82 const ( |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
83 createGeomTempTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
84 CREATE TEMP TABLE wp_geoms ( |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
85 geom geography(linestring, 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
86 ) ON COMMIT DROP` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
87 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
88 createTempIndexSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
89 CREATE INDEX ON wp_geoms USING GIST(geom)` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
90 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
91 analyzeTempTableSQL = `ANALYZE wp_geoms` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
92 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
93 insertGeomTmpTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
94 INSERT INTO wp_geoms (geom) VALUES ( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
95 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
96 )` |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
97 hasDistanceMarkSQL = ` |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
98 SELECT true FROM waterway.distance_marks_virtual |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
99 WHERE location_code = |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
100 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
101 LIMIT 1` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
102 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
103 insertWaterwayProfileSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
104 INSERT INTO waterway.waterway_profiles ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
105 location, |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
106 geom, |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
107 validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
108 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
109 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
110 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
111 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
112 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
113 date_info, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
114 source_organization |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
115 ) VALUES ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
116 ($1, $2, $3, $4, $5), |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
117 ( SELECT wp_geoms.geom |
2127
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
118 FROM wp_geoms, waterway.distance_marks_virtual AS dmv |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
119 WHERE dmv.location_code = |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
120 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
121 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
122 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true) |
2087
5d3d2e823314
Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2086
diff
changeset
|
123 LIMIT 1 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
124 ), |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
125 $6, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
126 $7, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
127 $8, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
128 $9, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
129 $10, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
130 $11, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
131 $12, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
132 $13 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
133 ) RETURNING id, geom is NULL` |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
134 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
135 wpStageDoneSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
136 UPDATE waterway.waterway_profiles SET staging_done = true |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
137 WHERE id IN ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
138 SELECT key FROM import.track_imports |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
139 WHERE import_id = $1 AND |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
140 relation = 'waterway.waterway_profiles'::regclass)` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
141 ) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
142 |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 func (wpJobCreator) StageDone( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
144 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
145 tx *sql.Tx, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
146 id int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
147 ) error { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
148 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
149 return err |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
150 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
151 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
152 func (wp *WaterwayProfiles) CleanUp() error { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 return os.RemoveAll(wp.Dir) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 func (wp *WaterwayProfiles) Do( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 importID int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 conn *sql.Conn, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 feedback Feedback, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 ) (interface{}, error) { |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
162 start := time.Now() |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
163 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
164 tx, err := conn.BeginTx(ctx, nil) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
165 if err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
166 return nil, err |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
167 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
168 defer tx.Rollback() |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
169 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
170 if err := wp.downloadGeometries( |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
171 ctx, importID, tx, start, feedback); err != nil { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
172 return nil, fmt.Errorf("error downloading geometries: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
173 } |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
174 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
175 summary, err := wp.processCSV( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
176 ctx, importID, tx, start, feedback) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
177 if err != nil { |
2143
f0641b5ad065
Waterway profile imports: Fixed typo in error message.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2127
diff
changeset
|
178 return nil, fmt.Errorf("error processing CSV: %v", err) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
179 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
180 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
181 if err := tx.Commit(); err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
182 return nil, fmt.Errorf( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
183 "Importing waterway profiles failed after %s: %v", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
184 time.Since(start), err) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
185 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
186 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
187 feedback.Info("Importing waterway profiles took %s", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
188 time.Since(start)) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
189 return summary, nil |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
190 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
191 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
192 func (wp *WaterwayProfiles) downloadGeometries( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
193 ctx context.Context, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
194 importID int64, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
195 tx *sql.Tx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
196 start time.Time, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
197 feedback Feedback, |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
198 ) error { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
199 feedback.Info("Start downloading geometries from WFS.") |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
200 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
201 feedback.Info("Loading capabilities from %s", wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
202 caps, err := wfs.GetCapabilities(wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
203 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
204 feedback.Error("Loading capabilities failed: %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
205 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
206 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
207 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
208 ft := caps.FindFeatureType(wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
209 if ft == nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
210 return fmt.Errorf("Unknown feature type '%s'", wp.FeatureType) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
211 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
212 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
213 feedback.Info("Found feature type '%s", wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
214 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
215 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
216 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
217 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
218 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
219 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
220 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
221 if wp.SortBy != "" { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
222 feedback.Info("Features will be sorted by '%s'", wp.SortBy) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
223 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
224 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
225 dl, err := wfs.GetFeatures(caps, wp.FeatureType, wp.SortBy) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
226 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
227 feedback.Error("Cannot create GetFeature URLs. %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
228 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
229 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
230 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
231 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
232 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
233 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
234 |
2126
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
235 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil { |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
236 return err |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
237 } |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
238 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
239 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
240 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
241 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
242 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
243 defer insertStmt.Close() |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
244 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
245 var ( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
246 unsupported = stringCounter{} |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
247 missingProperties int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
248 features int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
249 ) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
250 |
2729
7cb027be277d
WFS downloader: Really use username and password from parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2723
diff
changeset
|
251 if err := dl.Download(wp.User, wp.Password, func(url string, r io.Reader) error { |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2094
diff
changeset
|
252 feedback.Info("Get features from: '%s'", url) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
253 rfc, err := wfs.ParseRawFeatureCollection(r) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
254 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
255 return fmt.Errorf("parsing GetFeature document failed: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
256 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
257 if rfc.CRS != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
258 crsName := rfc.CRS.Properties.Name |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
259 if epsg, err = wfs.CRSToEPSG(crsName); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
260 feedback.Error("Unsupported CRS: %d", crsName) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
261 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
262 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
263 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
264 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
265 // No features -> ignore. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
266 if rfc.Features == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
267 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
268 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
269 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
270 for _, feature := range rfc.Features { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
271 if feature.Geometry.Coordinates == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
272 missingProperties++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
273 continue |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
274 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
275 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
276 switch feature.Geometry.Type { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
277 case "LineString": |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
278 var l lineSlice |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
279 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
280 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
281 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
282 if _, err := insertStmt.ExecContext( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
283 ctx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
284 l.asWKB(), |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
285 epsg, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
286 ); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
287 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
288 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
289 features++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
290 default: |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
291 unsupported[feature.Geometry.Type]++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
292 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
293 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
294 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
295 }); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
296 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
297 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
298 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
299 if missingProperties > 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
300 feedback.Warn("Missing properties: %d", missingProperties) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
301 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
302 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
303 if len(unsupported) != 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
304 feedback.Warn("Unsupported types found: %s", unsupported) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
305 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
306 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
307 if features == 0 { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
308 return errors.New("No features found") |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
309 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
310 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
311 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
312 } |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
313 return nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
314 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
315 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
316 func parseFloat64(s string) (sql.NullFloat64, error) { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
317 if s == "" { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
318 return sql.NullFloat64{}, nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
319 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
320 s = strings.Replace(s, ",", ".", -1) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
321 v, err := strconv.ParseFloat(s, 64) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
322 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
323 return sql.NullFloat64{}, err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
324 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
325 return sql.NullFloat64{Float64: v, Valid: true}, nil |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
326 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
327 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
328 func (wp *WaterwayProfiles) processCSV( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
329 ctx context.Context, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
330 importID int64, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
331 tx *sql.Tx, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
332 start time.Time, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
333 feedback Feedback, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
334 ) (interface{}, error) { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
335 feedback.Info("Start processing CSV file.") |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
336 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
337 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
338 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
339 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
340 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
341 defer f.Close() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
342 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
343 r := csv.NewReader(bufio.NewReader(f)) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
344 r.Comma = ';' |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
345 r.ReuseRecord = true |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
346 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
347 headers, err := r.Read() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
348 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
349 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
350 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
351 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
352 var ( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
353 locationIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
354 validFromIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
355 validToIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
356 lnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
357 mwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
358 hnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
359 fe30Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
360 fe100Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
361 dateInfoIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
362 sourceIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
363 ) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
364 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
365 fields := []struct { |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
366 idx *int |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
367 substr string |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
368 }{ |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
369 {&locationIdx, "location"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
370 {&validFromIdx, "valid_from"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
371 {&validToIdx, "valid_to"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
372 {&lnwlIdx, "lnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
373 {&mwlIdx, "mwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
374 {&hnwlIdx, "hnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
375 {&fe30Idx, "fe30"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
376 {&fe100Idx, "fe100"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
377 {&dateInfoIdx, "date_info"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
378 {&sourceIdx, "source"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
379 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
380 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
381 nextHeader: |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
382 for i, h := range headers { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
383 h = strings.ToLower(h) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
384 for j := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
385 if strings.Contains(h, fields[j].substr) { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
386 if *fields[j].idx != -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
387 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
388 "CSV has more than one column with name containing '%s'", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
389 fields[j].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
390 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
391 *fields[j].idx = i |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
392 continue nextHeader |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
393 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
394 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
395 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
396 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
397 var missing []string |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
398 for i := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
399 if *fields[i].idx == -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
400 missing = append(missing, fields[i].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
401 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
402 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
403 if len(missing) > 0 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
404 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
405 "CSV is missing columns: %s", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
406 strings.Join(missing, ", ")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
407 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
408 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
409 var precision float64 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
410 if wp.Precision != nil { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
411 if precision = *wp.Precision; precision < 0 { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
412 precision = -precision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
413 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
414 } else { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
415 precision = defaultPointToLinePrecision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
416 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
417 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
418 feedback.Info( |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
419 "Matching points to lines with a precision of %.4fm.", precision) |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
420 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
421 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
422 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
423 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
424 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
425 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
426 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
427 defer insertStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
428 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
429 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
430 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
431 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
432 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
433 defer trackStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
434 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
435 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
436 if err != nil { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
437 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
438 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
439 defer hasDistanceMarkStmt.Close() |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
440 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
441 var ids []int64 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
442 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
443 lines: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
444 for line := 1; ; line++ { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
445 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
446 row, err := r.Read() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
447 switch { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
448 case err == io.EOF || len(row) == 0: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
449 break lines |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
450 case err != nil: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
451 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
452 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
453 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
454 location, err := models.IsrsFromString(row[locationIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
455 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
456 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
457 "Invalid ISRS location code in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
458 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
459 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
460 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
461 var dummy bool |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
462 err = hasDistanceMarkStmt.QueryRowContext( |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
463 ctx, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
464 location.CountryCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
465 location.LoCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
466 location.FairwaySection, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
467 location.Orc, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
468 location.Hectometre, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
469 ).Scan(&dummy) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
470 |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
471 switch { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
472 case err == sql.ErrNoRows: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
473 feedback.Warn("No virtual distance mark found for %s.", location) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
474 continue lines |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
475 case err != nil: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
476 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
477 case !dummy: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
478 return nil, errors.New("unexpected result form database") |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
479 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
480 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
481 validFromTime, err := parseDate(row[validFromIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
482 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
483 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
484 "Invalid 'valid_from' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
485 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
486 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
487 validToTime, err := parseDate(row[validToIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
488 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
489 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
490 "Invalid 'valid_to' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
491 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
492 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
493 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
494 validFrom := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
495 Time: validFromTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
496 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
497 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
498 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
499 validTo := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
500 Time: validToTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
501 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
502 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
503 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
504 validity := pgtype.Tstzrange{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
505 Lower: validFrom, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
506 Upper: validTo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
507 LowerType: pgtype.Inclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
508 UpperType: pgtype.Exclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
509 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
510 } |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
511 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
512 lnwl, err := parseFloat64(row[lnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
513 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
514 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
515 "Invalid 'lnwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
516 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
517 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
518 mwl, err := parseFloat64(row[mwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
519 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
520 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
521 "Invalid 'mwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
522 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
523 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
524 hnwl, err := parseFloat64(row[hnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
525 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
526 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
527 "Invalid 'hnwl' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
528 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
529 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
530 fe30, err := parseFloat64(row[fe30Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
531 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
532 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
533 "Invalid 'fe30' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
534 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
535 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
536 fe100, err := parseFloat64(row[fe100Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
537 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
538 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
539 "Invalid 'fe100' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
540 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
541 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
542 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
543 var dateInfo time.Time |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
544 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
545 if di := row[dateInfoIdx]; di == "" { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
546 dateInfo = start |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
547 } else if dateInfo, err = parseDate(di); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
548 return nil, fmt.Errorf( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
549 "Invalid 'date_info' value in line %d: %v", |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
550 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
551 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
552 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
553 source := row[sourceIdx] |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
554 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
555 var id int64 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
556 var noGeom bool |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
557 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
558 if err := insertStmt.QueryRowContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
559 ctx, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
560 location.CountryCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
561 location.LoCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
562 location.FairwaySection, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
563 location.Orc, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
564 location.Hectometre, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
565 &validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
566 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
567 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
568 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
569 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
570 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
571 dateInfo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
572 source, |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
573 precision, |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
574 ).Scan(&id, &noGeom); err != nil { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
575 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
576 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
577 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
578 if _, err := trackStmt.ExecContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
579 ctx, importID, "waterway.waterway_profiles", id); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
580 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
581 } |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
582 |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
583 if noGeom { |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
584 feedback.Warn( |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
585 "No profile geometry found for %s in line %d.", location, line) |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
586 } |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
587 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
588 ids = append(ids, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
589 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
590 if len(ids) == 0 { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
591 return nil, UnchangedError("No new entries in waterway profiles.") |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
592 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
593 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
594 feedback.Info("%d new entries in waterway profiles.", len(ids)) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
595 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
596 summary := struct { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
597 IDs []int64 `json:"ids"` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
598 }{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
599 IDs: ids, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
600 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
601 return &summary, nil |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
602 } |