Mercurial > gemma
annotate pkg/imports/wp.go @ 4855:01b593ea2311
Added missing doc strings for the STR tree in the mesh package.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Tue, 19 Nov 2019 17:48:02 +0100 |
parents | 046a07a33b19 |
children | 783d0bec86d3 |
rev | line source |
---|---|
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
17 "bufio" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
20 "encoding/csv" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
21 "encoding/json" |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
22 "errors" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
23 "fmt" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
24 "io" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "os" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
26 "path/filepath" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
27 "strconv" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
28 "strings" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
29 "time" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
31 "github.com/jackc/pgx/pgtype" |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
32 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
33 "gemma.intevation.de/gemma/pkg/common" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/models" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/wfs" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
38 // defaultPointToLinePrecision is the precision in meters |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
39 // to match from points to lines. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
40 const defaultPointToLinePrecision = 10 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
41 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
42 // WaterwayProfiles is a Job to import waterway profiles |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
43 // from a given WFS service plus some uploaded CSV file |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
44 // and stores them into the database. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 type WaterwayProfiles struct { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 Dir string `json:"dir"` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
47 // URL the GetCapabilities URL of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
48 URL string `json:"url"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
49 // FeatureType selects the feature type of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
50 FeatureType string `json:"feature-type"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
51 // SortBy works around misconfigured services to |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
52 // establish a sort order to get the features. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
53 SortBy string `json:"sort-by"` |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
54 // Precsion of match points to line strings. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
55 Precision *float64 `json:"precision,omitempty"` |
2723
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
56 // User is an optional username for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
57 User string `json:"user,omitempty"` |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
58 // Password is an optional password for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
59 Password string `json:"password,omitempty"` |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
4852
046a07a33b19
Fixed the golint issues of the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4832
diff
changeset
|
62 // Description gives a short info about relevant facts of this import. |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4182
diff
changeset
|
63 func (wp *WaterwayProfiles) Description() (string, error) { |
4799
f32d086b5dbf
Removed the mechanical touch of the last commit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4798
diff
changeset
|
64 return wp.URL + "|" + wp.FeatureType, nil |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4182
diff
changeset
|
65 } |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4182
diff
changeset
|
66 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
67 // WPJobKind is the unique name of this import job type. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 const WPJobKind JobKind = "wp" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 type wpJobCreator struct{} |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 func init() { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 RegisterJobCreator(WPJobKind, wpJobCreator{}) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2183
diff
changeset
|
76 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 func (wpJobCreator) AutoAccept() bool { return false } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 func (wpJobCreator) Description() string { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 return "waterway profiles" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
84 func (wpJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
85 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
86 {"waterway_profiles"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
87 {"distance_marks_virtual"}, |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
91 const ( |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
92 createGeomTempTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
93 CREATE TEMP TABLE wp_geoms ( |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
94 geom geography(linestring, 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
95 ) ON COMMIT DROP` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
96 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
97 createTempIndexSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
98 CREATE INDEX ON wp_geoms USING GIST(geom)` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
99 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
100 analyzeTempTableSQL = `ANALYZE wp_geoms` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
101 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
102 insertGeomTmpTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
103 INSERT INTO wp_geoms (geom) VALUES ( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
104 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
105 )` |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
106 hasDistanceMarkSQL = ` |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
107 SELECT true FROM waterway.distance_marks_virtual |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
108 WHERE location_code = |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
109 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
110 LIMIT 1` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
111 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
112 insertWaterwayProfileSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
113 INSERT INTO waterway.waterway_profiles ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
114 location, |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
115 geom, |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
116 validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
117 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
118 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
119 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
120 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
121 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
122 date_info, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
123 source_organization |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
124 ) VALUES ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
125 ($1, $2, $3, $4, $5), |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
126 ( SELECT wp_geoms.geom |
2127
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
127 FROM wp_geoms, waterway.distance_marks_virtual AS dmv |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
128 WHERE dmv.location_code = |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
129 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
130 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
131 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true) |
2087
5d3d2e823314
Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2086
diff
changeset
|
132 LIMIT 1 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
133 ), |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
134 $6, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
135 $7, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
136 $8, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
137 $9, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
138 $10, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
139 $11, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
140 $12, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
141 $13 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
142 ) RETURNING id, geom is NULL` |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
143 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
144 wpStageDoneSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
145 UPDATE waterway.waterway_profiles SET staging_done = true |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
146 WHERE id IN ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
147 SELECT key FROM import.track_imports |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
148 WHERE import_id = $1 AND |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
149 relation = 'waterway.waterway_profiles'::regclass)` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
150 ) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
151 |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
152 func (wpJobCreator) StageDone( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 tx *sql.Tx, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 id int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 ) error { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
157 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
158 return err |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
161 // CleanUp deletes temporary files from the filesystem. |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
162 func (wp *WaterwayProfiles) CleanUp() error { return os.RemoveAll(wp.Dir) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
164 // Do performs the actual import. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 func (wp *WaterwayProfiles) Do( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 importID int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 conn *sql.Conn, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 feedback Feedback, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 ) (interface{}, error) { |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
171 start := time.Now() |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
172 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
173 tx, err := conn.BeginTx(ctx, nil) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
174 if err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
175 return nil, err |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
176 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
177 defer tx.Rollback() |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
178 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
179 if err := wp.downloadGeometries( |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
180 ctx, importID, tx, start, feedback); err != nil { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
181 return nil, fmt.Errorf("error downloading geometries: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
182 } |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
183 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
184 summary, err := wp.processCSV( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
185 ctx, importID, tx, start, feedback) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
186 if err != nil { |
2143
f0641b5ad065
Waterway profile imports: Fixed typo in error message.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2127
diff
changeset
|
187 return nil, fmt.Errorf("error processing CSV: %v", err) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
188 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
189 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
190 if err := tx.Commit(); err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
191 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
192 "importing waterway profiles failed after %s: %v", |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
193 time.Since(start), err) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
194 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
195 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
196 feedback.Info("Importing waterway profiles took %s", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
197 time.Since(start)) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
198 return summary, nil |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
199 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
200 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
201 func (wp *WaterwayProfiles) downloadGeometries( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
202 ctx context.Context, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
203 importID int64, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
204 tx *sql.Tx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
205 start time.Time, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
206 feedback Feedback, |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
207 ) error { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
208 feedback.Info("Start downloading geometries from WFS.") |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
209 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
210 feedback.Info("Loading capabilities from %s", wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
211 caps, err := wfs.GetCapabilities(wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
212 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
213 feedback.Error("Loading capabilities failed: %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
214 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
215 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
216 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
217 ft := caps.FindFeatureType(wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
218 if ft == nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
219 return fmt.Errorf("unknown feature type '%s'", wp.FeatureType) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
220 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
221 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
222 feedback.Info("Found feature type '%s", wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
223 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
224 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
225 if err != nil { |
4832
5062ccb2381d
Unified error message on "Unsupported CRS"
Sascha Wilde <wilde@intevation.de>
parents:
4799
diff
changeset
|
226 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
227 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
228 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
229 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
230 if wp.SortBy != "" { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
231 feedback.Info("Features will be sorted by '%s'", wp.SortBy) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
232 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
233 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
234 dl, err := wfs.GetFeatures(caps, wp.FeatureType, wp.SortBy) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
235 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
236 feedback.Error("Cannot create GetFeature URLs. %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
237 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
238 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
239 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
240 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
241 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
242 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
243 |
2126
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
244 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil { |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
245 return err |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
246 } |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
247 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
248 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
249 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
250 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
251 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
252 defer insertStmt.Close() |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
253 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
254 var ( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
255 unsupported = stringCounter{} |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
256 missingProperties int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
257 features int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
258 ) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
259 |
2729
7cb027be277d
WFS downloader: Really use username and password from parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2723
diff
changeset
|
260 if err := dl.Download(wp.User, wp.Password, func(url string, r io.Reader) error { |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2094
diff
changeset
|
261 feedback.Info("Get features from: '%s'", url) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
262 rfc, err := wfs.ParseRawFeatureCollection(r) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
263 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
264 return fmt.Errorf("parsing GetFeature document failed: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
265 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
266 if rfc.CRS != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
267 crsName := rfc.CRS.Properties.Name |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
268 if epsg, err = wfs.CRSToEPSG(crsName); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
269 feedback.Error("Unsupported CRS: %d", crsName) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
270 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
271 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
272 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
273 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
274 // No features -> ignore. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
275 if rfc.Features == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
276 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
277 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
278 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
279 for _, feature := range rfc.Features { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
280 if feature.Geometry.Coordinates == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
281 missingProperties++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
282 continue |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
283 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
284 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
285 switch feature.Geometry.Type { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
286 case "LineString": |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
287 var l lineSlice |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
288 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
289 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
290 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
291 if _, err := insertStmt.ExecContext( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
292 ctx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
293 l.asWKB(), |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
294 epsg, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
295 ); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
296 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
297 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
298 features++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
299 default: |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
300 unsupported[feature.Geometry.Type]++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
301 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
302 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
303 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
304 }); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
305 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
306 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
307 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
308 if missingProperties > 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
309 feedback.Warn("Missing properties: %d", missingProperties) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
310 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
311 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
312 if len(unsupported) != 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
313 feedback.Warn("Unsupported types found: %s", unsupported) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
314 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
315 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
316 if features == 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
317 return errors.New("no features found") |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
318 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
319 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
320 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
321 } |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
322 return nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
323 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
324 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
325 func parseFloat64(s string) (sql.NullFloat64, error) { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
326 if s == "" { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
327 return sql.NullFloat64{}, nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
328 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
329 s = strings.Replace(s, ",", ".", -1) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
330 v, err := strconv.ParseFloat(s, 64) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
331 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
332 return sql.NullFloat64{}, err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
333 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
334 return sql.NullFloat64{Float64: v, Valid: true}, nil |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
335 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
336 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
337 func (wp *WaterwayProfiles) processCSV( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
338 ctx context.Context, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
339 importID int64, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
340 tx *sql.Tx, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
341 start time.Time, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
342 feedback Feedback, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
343 ) (interface{}, error) { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
344 feedback.Info("Start processing CSV file.") |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
345 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
346 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
347 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
348 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
349 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
350 defer f.Close() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
351 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
352 r := csv.NewReader(bufio.NewReader(f)) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
353 r.Comma = ';' |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
354 r.ReuseRecord = true |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
355 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
356 headers, err := r.Read() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
357 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
358 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
359 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
360 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
361 var ( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
362 locationIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
363 validFromIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
364 validToIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
365 lnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
366 mwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
367 hnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
368 fe30Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
369 fe100Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
370 dateInfoIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
371 sourceIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
372 ) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
373 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
374 fields := []struct { |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
375 idx *int |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
376 substr string |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
377 }{ |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
378 {&locationIdx, "location"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
379 {&validFromIdx, "valid_from"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
380 {&validToIdx, "valid_to"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
381 {&lnwlIdx, "lnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
382 {&mwlIdx, "mwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
383 {&hnwlIdx, "hnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
384 {&fe30Idx, "fe30"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
385 {&fe100Idx, "fe100"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
386 {&dateInfoIdx, "date_info"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
387 {&sourceIdx, "source"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
388 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
389 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
390 nextHeader: |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
391 for i, h := range headers { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
392 h = strings.ToLower(h) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
393 for j := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
394 if strings.Contains(h, fields[j].substr) { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
395 if *fields[j].idx != -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
396 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
397 "CSV has more than one column with name containing '%s'", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
398 fields[j].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
399 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
400 *fields[j].idx = i |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
401 continue nextHeader |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
402 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
403 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
404 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
405 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
406 var missing []string |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
407 for i := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
408 if *fields[i].idx == -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
409 missing = append(missing, fields[i].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
410 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
411 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
412 if len(missing) > 0 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
413 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
414 "CSV is missing columns: %s", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
415 strings.Join(missing, ", ")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
416 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
417 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
418 var precision float64 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
419 if wp.Precision != nil { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
420 if precision = *wp.Precision; precision < 0 { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
421 precision = -precision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
422 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
423 } else { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
424 precision = defaultPointToLinePrecision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
425 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
426 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
427 feedback.Info( |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
428 "Matching points to lines with a precision of %.4fm.", precision) |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
429 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
430 parseDate := common.TimeParser([]string{"02.01.2006"}).Parse |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
431 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
432 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
433 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
434 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
435 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
436 defer insertStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
437 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
438 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
439 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
440 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
441 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
442 defer trackStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
443 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
444 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
445 if err != nil { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
446 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
447 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
448 defer hasDistanceMarkStmt.Close() |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
449 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
450 var ids []int64 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
451 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
452 lines: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
453 for line := 1; ; line++ { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
454 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
455 row, err := r.Read() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
456 switch { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
457 case err == io.EOF || len(row) == 0: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
458 break lines |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
459 case err != nil: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
460 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
461 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
462 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
463 location, err := models.IsrsFromString(row[locationIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
464 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
465 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
466 "invalid ISRS location code in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
467 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
468 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
469 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
470 var dummy bool |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
471 err = hasDistanceMarkStmt.QueryRowContext( |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
472 ctx, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
473 location.CountryCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
474 location.LoCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
475 location.FairwaySection, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
476 location.Orc, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
477 location.Hectometre, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
478 ).Scan(&dummy) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
479 |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
480 switch { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
481 case err == sql.ErrNoRows: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
482 feedback.Warn("No virtual distance mark found for %s.", location) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
483 continue lines |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
484 case err != nil: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
485 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
486 case !dummy: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
487 return nil, errors.New("unexpected result form database") |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
488 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
489 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
490 validFromTime, err := parseDate(row[validFromIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
491 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
492 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
493 "invalid 'valid_from' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
494 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
495 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
496 validToTime, err := parseDate(row[validToIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
497 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
498 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
499 "invalid 'valid_to' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
500 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
501 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
502 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
503 validFrom := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
504 Time: validFromTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
505 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
506 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
507 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
508 validTo := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
509 Time: validToTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
510 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
511 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
512 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
513 validity := pgtype.Tstzrange{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
514 Lower: validFrom, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
515 Upper: validTo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
516 LowerType: pgtype.Inclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
517 UpperType: pgtype.Exclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
518 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
519 } |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
520 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
521 lnwl, err := parseFloat64(row[lnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
522 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
523 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
524 "invalid 'lnwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
525 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
526 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
527 mwl, err := parseFloat64(row[mwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
528 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
529 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
530 "invalid 'mwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
531 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
532 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
533 hnwl, err := parseFloat64(row[hnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
534 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
535 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
536 "invalid 'hnwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
537 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
538 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
539 fe30, err := parseFloat64(row[fe30Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
540 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
541 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
542 "invalid 'fe30' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
543 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
544 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
545 fe100, err := parseFloat64(row[fe100Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
546 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
547 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
548 "invalid 'fe100' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
549 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
550 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
551 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
552 var dateInfo time.Time |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
553 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
554 if di := row[dateInfoIdx]; di == "" { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
555 dateInfo = start |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
556 } else if dateInfo, err = parseDate(di); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
557 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
558 "invalid 'date_info' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
559 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
560 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
561 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
562 source := row[sourceIdx] |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
563 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
564 var id int64 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
565 var noGeom bool |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
566 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
567 if err := insertStmt.QueryRowContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
568 ctx, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
569 location.CountryCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
570 location.LoCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
571 location.FairwaySection, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
572 location.Orc, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
573 location.Hectometre, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
574 &validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
575 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
576 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
577 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
578 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
579 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
580 dateInfo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
581 source, |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
582 precision, |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
583 ).Scan(&id, &noGeom); err != nil { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
584 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
585 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
586 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
587 if _, err := trackStmt.ExecContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
588 ctx, importID, "waterway.waterway_profiles", id); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
589 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
590 } |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
591 |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
592 if noGeom { |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
593 feedback.Warn( |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
594 "No profile geometry found for %s in line %d.", location, line) |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
595 } |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
596 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
597 ids = append(ids, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
598 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
599 if len(ids) == 0 { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
600 return nil, UnchangedError("No new entries in waterway profiles.") |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
601 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
602 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
603 feedback.Info("%d new entries in waterway profiles.", len(ids)) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
604 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
605 summary := struct { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
606 IDs []int64 `json:"ids"` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
607 }{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
608 IDs: ids, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
609 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
610 return &summary, nil |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
611 } |