Mercurial > gemma
annotate pkg/imports/wp.go @ 5591:0011f50cf216 surveysperbottleneckid
Removed no longer used alternative api for surveys/ endpoint.
As bottlenecks in the summary for SR imports are now identified by
their id and no longer by the (not guarantied to be unique!) name,
there is no longer the need to request survey data by the name+date
tuple (which isn't reliable anyway). So the workaround was now
reversed.
author | Sascha Wilde <wilde@sha-bang.de> |
---|---|
date | Wed, 06 Apr 2022 13:30:29 +0200 |
parents | ade07a3f2cfd |
children | 2dd155cc95ec |
rev | line source |
---|---|
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
17 "bufio" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
20 "encoding/csv" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
21 "encoding/json" |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
22 "errors" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
23 "fmt" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
24 "io" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "os" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
26 "path/filepath" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
27 "strconv" |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
28 "strings" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
29 "time" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
31 "github.com/jackc/pgx/pgtype" |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
32 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
33 "gemma.intevation.de/gemma/pkg/common" |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/models" |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/wfs" |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
38 // defaultPointToLinePrecision is the precision in meters |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
39 // to match from points to lines. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
40 const defaultPointToLinePrecision = 10 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
41 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
42 // WaterwayProfiles is a Job to import waterway profiles |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
43 // from a given WFS service plus some uploaded CSV file |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
44 // and stores them into the database. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 type WaterwayProfiles struct { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 Dir string `json:"dir"` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
47 // URL the GetCapabilities URL of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
48 URL string `json:"url"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
49 // FeatureType selects the feature type of the WFS service. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
50 FeatureType string `json:"feature-type"` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
51 // SortBy works around misconfigured services to |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
52 // establish a sort order to get the features. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
53 SortBy string `json:"sort-by"` |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
54 // Precsion of match points to line strings. |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
55 Precision *float64 `json:"precision,omitempty"` |
2723
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
56 // User is an optional username for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
57 User string `json:"user,omitempty"` |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
58 // Password is an optional password for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
59 Password string `json:"password,omitempty"` |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
4852
046a07a33b19
Fixed the golint issues of the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4832
diff
changeset
|
62 // Description gives a short info about relevant facts of this import. |
5565
ade07a3f2cfd
Forget to change signature of some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5270
diff
changeset
|
63 func (wp *WaterwayProfiles) Description([]string) (string, error) { |
4799
f32d086b5dbf
Removed the mechanical touch of the last commit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4798
diff
changeset
|
64 return wp.URL + "|" + wp.FeatureType, nil |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4182
diff
changeset
|
65 } |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4182
diff
changeset
|
66 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
67 // WPJobKind is the unique name of this import job type. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 const WPJobKind JobKind = "wp" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 type wpJobCreator struct{} |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 func init() { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 RegisterJobCreator(WPJobKind, wpJobCreator{}) |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2183
diff
changeset
|
76 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 func (wpJobCreator) AutoAccept() bool { return false } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 func (wpJobCreator) Description() string { |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 return "waterway profiles" |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
84 func (wpJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
85 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
86 {"waterway_profiles"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2729
diff
changeset
|
87 {"distance_marks_virtual"}, |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
91 const ( |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
92 createGeomTempTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
93 CREATE TEMP TABLE wp_geoms ( |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
94 geom geography(linestring, 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
95 ) ON COMMIT DROP` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
96 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
97 createTempIndexSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
98 CREATE INDEX ON wp_geoms USING GIST(geom)` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
99 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
100 analyzeTempTableSQL = `ANALYZE wp_geoms` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
101 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
102 insertGeomTmpTableSQL = ` |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
103 INSERT INTO wp_geoms (geom) VALUES ( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
104 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
105 )` |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
106 hasDistanceMarkSQL = ` |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
107 SELECT true FROM waterway.distance_marks_virtual |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
108 WHERE location_code = |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
109 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
110 LIMIT 1` |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
111 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
112 insertWaterwayProfileSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
113 INSERT INTO waterway.waterway_profiles ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
114 location, |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
115 geom, |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
116 validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
117 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
118 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
119 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
120 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
121 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
122 date_info, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
123 source_organization |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
124 ) VALUES ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
125 ($1, $2, $3, $4, $5), |
2093
a9d28c489761
Waterway profile import: Do all the calculations in geography domain.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2092
diff
changeset
|
126 ( SELECT wp_geoms.geom |
2127
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
127 FROM wp_geoms, waterway.distance_marks_virtual AS dmv |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
128 WHERE dmv.location_code = |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
129 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
130 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float) |
a1f2cfa624cf
Make distance filtering more consistent
Tom Gottfried <tom@intevation.de>
parents:
2126
diff
changeset
|
131 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true) |
2087
5d3d2e823314
Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2086
diff
changeset
|
132 LIMIT 1 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
133 ), |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
134 $6, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
135 $7, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
136 $8, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
137 $9, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
138 $10, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
139 $11, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
140 $12, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
141 $13 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
142 ) RETURNING id, geom is NULL` |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
143 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
144 wpStageDoneSQL = ` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
145 UPDATE waterway.waterway_profiles SET staging_done = true |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
146 WHERE id IN ( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
147 SELECT key FROM import.track_imports |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
148 WHERE import_id = $1 AND |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
149 relation = 'waterway.waterway_profiles'::regclass)` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
150 ) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
151 |
5035
56c589f7435d
Enhance comments on implementations of StageDone()
Tom Gottfried <tom@intevation.de>
parents:
5034
diff
changeset
|
152 // StageDone moves the imported waterway profiles out of the staging area. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 func (wpJobCreator) StageDone( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 tx *sql.Tx, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 id int64, |
5034
59a99655f34d
Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4879
diff
changeset
|
157 _ Feedback, |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 ) error { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
159 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
160 return err |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 } |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
163 // CleanUp deletes temporary files from the filesystem. |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
164 func (wp *WaterwayProfiles) CleanUp() error { return os.RemoveAll(wp.Dir) } |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
166 // Do performs the actual import. |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 func (wp *WaterwayProfiles) Do( |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 ctx context.Context, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 importID int64, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 conn *sql.Conn, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 feedback Feedback, |
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 ) (interface{}, error) { |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
173 start := time.Now() |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
174 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
175 tx, err := conn.BeginTx(ctx, nil) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
176 if err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
177 return nil, err |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
178 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
179 defer tx.Rollback() |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
180 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
181 if err := wp.downloadGeometries( |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
182 ctx, importID, tx, start, feedback); err != nil { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
183 return nil, fmt.Errorf("error downloading geometries: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
184 } |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
185 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
186 summary, err := wp.processCSV( |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
187 ctx, importID, tx, start, feedback) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
188 if err != nil { |
2143
f0641b5ad065
Waterway profile imports: Fixed typo in error message.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2127
diff
changeset
|
189 return nil, fmt.Errorf("error processing CSV: %v", err) |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
190 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
191 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
192 if err := tx.Commit(); err != nil { |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
193 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
194 "importing waterway profiles failed after %s: %v", |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
195 time.Since(start), err) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
196 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
197 |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
198 feedback.Info("Importing waterway profiles took %s", |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
199 time.Since(start)) |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
200 return summary, nil |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
201 } |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
202 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
203 func (wp *WaterwayProfiles) downloadGeometries( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
204 ctx context.Context, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
205 importID int64, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
206 tx *sql.Tx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
207 start time.Time, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
208 feedback Feedback, |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
209 ) error { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
210 feedback.Info("Start downloading geometries from WFS.") |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
211 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
212 feedback.Info("Loading capabilities from %s", wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
213 caps, err := wfs.GetCapabilities(wp.URL) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
214 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
215 feedback.Error("Loading capabilities failed: %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
216 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
217 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
218 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
219 ft := caps.FindFeatureType(wp.FeatureType) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
220 if ft == nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
221 return fmt.Errorf("unknown feature type '%s'", wp.FeatureType) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
222 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
223 |
4879
783d0bec86d3
Add missing quotation marks
Tom Gottfried <tom@intevation.de>
parents:
4852
diff
changeset
|
224 feedback.Info("Found feature type '%s'", wp.FeatureType) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
225 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
226 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
227 if err != nil { |
4832
5062ccb2381d
Unified error message on "Unsupported CRS"
Sascha Wilde <wilde@intevation.de>
parents:
4799
diff
changeset
|
228 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
229 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
230 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
231 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
232 if wp.SortBy != "" { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
233 feedback.Info("Features will be sorted by '%s'", wp.SortBy) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
234 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
235 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
236 dl, err := wfs.GetFeatures(caps, wp.FeatureType, wp.SortBy) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
237 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
238 feedback.Error("Cannot create GetFeature URLs. %v", err) |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
239 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
240 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
241 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
242 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
243 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
244 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
245 |
2126
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
246 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil { |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
247 return err |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
248 } |
d4a5e475f874
Actually create index on temp table
Tom Gottfried <tom@intevation.de>
parents:
2109
diff
changeset
|
249 |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
250 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
251 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
252 return err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
253 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
254 defer insertStmt.Close() |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
255 |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
256 var ( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
257 unsupported = stringCounter{} |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
258 missingProperties int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
259 features int |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
260 ) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
261 |
2729
7cb027be277d
WFS downloader: Really use username and password from parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2723
diff
changeset
|
262 if err := dl.Download(wp.User, wp.Password, func(url string, r io.Reader) error { |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2094
diff
changeset
|
263 feedback.Info("Get features from: '%s'", url) |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
264 rfc, err := wfs.ParseRawFeatureCollection(r) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
265 if err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
266 return fmt.Errorf("parsing GetFeature document failed: %v", err) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
267 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
268 if rfc.CRS != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
269 crsName := rfc.CRS.Properties.Name |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
270 if epsg, err = wfs.CRSToEPSG(crsName); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
271 feedback.Error("Unsupported CRS: %d", crsName) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
272 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
273 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
274 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
275 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
276 // No features -> ignore. |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
277 if rfc.Features == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
278 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
279 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
280 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
281 for _, feature := range rfc.Features { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
282 if feature.Geometry.Coordinates == nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
283 missingProperties++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
284 continue |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
285 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
286 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
287 switch feature.Geometry.Type { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
288 case "LineString": |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
289 var l lineSlice |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
290 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
291 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
292 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
293 if _, err := insertStmt.ExecContext( |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
294 ctx, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
295 l.asWKB(), |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
296 epsg, |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
297 ); err != nil { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
298 return err |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
299 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
300 features++ |
5270
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
301 case "MultiLineString": |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
302 var ml multiLineSlice |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
303 if err := json.Unmarshal(*feature.Geometry.Coordinates, &ml); err != nil { |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
304 return err |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
305 } |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
306 for i := range ml { |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
307 if _, err := insertStmt.ExecContext( |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
308 ctx, |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
309 ml[i].asWKB(), |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
310 epsg, |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
311 ); err != nil { |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
312 return err |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
313 } |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
314 features++ |
41a67619c170
Added multi geometry support for some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5035
diff
changeset
|
315 } |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
316 default: |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
317 unsupported[feature.Geometry.Type]++ |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
318 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
319 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
320 return nil |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
321 }); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
322 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
323 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
324 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
325 if missingProperties > 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
326 feedback.Warn("Missing properties: %d", missingProperties) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
327 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
328 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
329 if len(unsupported) != 0 { |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
330 feedback.Warn("Unsupported types found: %s", unsupported) |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
331 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
332 |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
333 if features == 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
334 return errors.New("no features found") |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
335 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
336 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil { |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
337 return err |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
338 } |
2091
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
339 return nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
340 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
341 |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
342 func parseFloat64(s string) (sql.NullFloat64, error) { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
343 if s == "" { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
344 return sql.NullFloat64{}, nil |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
345 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
346 s = strings.Replace(s, ",", ".", -1) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
347 v, err := strconv.ParseFloat(s, 64) |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
348 if err != nil { |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
349 return sql.NullFloat64{}, err |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
350 } |
3f32b18bbd4c
Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2087
diff
changeset
|
351 return sql.NullFloat64{Float64: v, Valid: true}, nil |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
352 } |
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
353 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
354 func (wp *WaterwayProfiles) processCSV( |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
355 ctx context.Context, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
356 importID int64, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
357 tx *sql.Tx, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
358 start time.Time, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
359 feedback Feedback, |
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
360 ) (interface{}, error) { |
2086
6096ec4951f8
Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2084
diff
changeset
|
361 feedback.Info("Start processing CSV file.") |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
362 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
363 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
364 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
365 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
366 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
367 defer f.Close() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
368 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
369 r := csv.NewReader(bufio.NewReader(f)) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
370 r.Comma = ';' |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
371 r.ReuseRecord = true |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
372 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
373 headers, err := r.Read() |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
374 if err != nil { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
375 return nil, err |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
376 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
377 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
378 var ( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
379 locationIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
380 validFromIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
381 validToIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
382 lnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
383 mwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
384 hnwlIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
385 fe30Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
386 fe100Idx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
387 dateInfoIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
388 sourceIdx = -1 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
389 ) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
390 |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
391 fields := []struct { |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
392 idx *int |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
393 substr string |
2084
ddbac0f22ffb
Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2079
diff
changeset
|
394 }{ |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
395 {&locationIdx, "location"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
396 {&validFromIdx, "valid_from"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
397 {&validToIdx, "valid_to"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
398 {&lnwlIdx, "lnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
399 {&mwlIdx, "mwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
400 {&hnwlIdx, "hnwl"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
401 {&fe30Idx, "fe30"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
402 {&fe100Idx, "fe100"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
403 {&dateInfoIdx, "date_info"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
404 {&sourceIdx, "source"}, |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
405 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
406 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
407 nextHeader: |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
408 for i, h := range headers { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
409 h = strings.ToLower(h) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
410 for j := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
411 if strings.Contains(h, fields[j].substr) { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
412 if *fields[j].idx != -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
413 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
414 "CSV has more than one column with name containing '%s'", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
415 fields[j].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
416 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
417 *fields[j].idx = i |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
418 continue nextHeader |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
419 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
420 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
421 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
422 |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
423 var missing []string |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
424 for i := range fields { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
425 if *fields[i].idx == -1 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
426 missing = append(missing, fields[i].substr) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
427 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
428 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
429 if len(missing) > 0 { |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
430 return nil, fmt.Errorf( |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
431 "CSV is missing columns: %s", |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
432 strings.Join(missing, ", ")) |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
433 } |
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
434 |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
435 var precision float64 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
436 if wp.Precision != nil { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
437 if precision = *wp.Precision; precision < 0 { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
438 precision = -precision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
439 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
440 } else { |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
441 precision = defaultPointToLinePrecision |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
442 } |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
443 |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
444 feedback.Info( |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
445 "Matching points to lines with a precision of %.4fm.", precision) |
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
446 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
447 parseDate := common.TimeParser([]string{"02.01.2006"}).Parse |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
448 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
449 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
450 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
451 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
452 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
453 defer insertStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
454 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
455 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
456 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
457 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
458 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
459 defer trackStmt.Close() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
460 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
461 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
462 if err != nil { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
463 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
464 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
465 defer hasDistanceMarkStmt.Close() |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
466 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
467 var ids []int64 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
468 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
469 lines: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
470 for line := 1; ; line++ { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
471 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
472 row, err := r.Read() |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
473 switch { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
474 case err == io.EOF || len(row) == 0: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
475 break lines |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
476 case err != nil: |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
477 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
478 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
479 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
480 location, err := models.IsrsFromString(row[locationIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
481 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
482 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
483 "invalid ISRS location code in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
484 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
485 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
486 |
2183
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
487 var dummy bool |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
488 err = hasDistanceMarkStmt.QueryRowContext( |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
489 ctx, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
490 location.CountryCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
491 location.LoCode, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
492 location.FairwaySection, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
493 location.Orc, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
494 location.Hectometre, |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
495 ).Scan(&dummy) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
496 |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
497 switch { |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
498 case err == sql.ErrNoRows: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
499 feedback.Warn("No virtual distance mark found for %s.", location) |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
500 continue lines |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
501 case err != nil: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
502 return nil, err |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
503 case !dummy: |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
504 return nil, errors.New("unexpected result form database") |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
505 } |
b27edd1f2353
Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
506 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
507 validFromTime, err := parseDate(row[validFromIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
508 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
509 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
510 "invalid 'valid_from' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
511 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
512 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
513 validToTime, err := parseDate(row[validToIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
514 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
515 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
516 "invalid 'valid_to' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
517 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
518 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
519 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
520 validFrom := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
521 Time: validFromTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
522 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
523 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
524 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
525 validTo := pgtype.Timestamptz{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
526 Time: validToTime, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
527 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
528 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
529 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
530 validity := pgtype.Tstzrange{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
531 Lower: validFrom, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
532 Upper: validTo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
533 LowerType: pgtype.Inclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
534 UpperType: pgtype.Exclusive, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
535 Status: pgtype.Present, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
536 } |
2078
dacf79a0658e
Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2072
diff
changeset
|
537 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
538 lnwl, err := parseFloat64(row[lnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
539 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
540 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
541 "invalid 'lnwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
542 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
543 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
544 mwl, err := parseFloat64(row[mwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
545 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
546 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
547 "invalid 'mwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
548 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
549 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
550 hnwl, err := parseFloat64(row[hnwlIdx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
551 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
552 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
553 "invalid 'hnwl' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
554 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
555 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
556 fe30, err := parseFloat64(row[fe30Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
557 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
558 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
559 "invalid 'fe30' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
560 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
561 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
562 fe100, err := parseFloat64(row[fe100Idx]) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
563 if err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
564 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
565 "invalid 'fe100' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
566 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
567 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
568 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
569 var dateInfo time.Time |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
570 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
571 if di := row[dateInfoIdx]; di == "" { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
572 dateInfo = start |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
573 } else if dateInfo, err = parseDate(di); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
574 return nil, fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4075
diff
changeset
|
575 "invalid 'date_info' value in line %d: %v", |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
576 line, err) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
577 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
578 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
579 source := row[sourceIdx] |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
580 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
581 var id int64 |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
582 var noGeom bool |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
583 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
584 if err := insertStmt.QueryRowContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
585 ctx, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
586 location.CountryCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
587 location.LoCode, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
588 location.FairwaySection, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
589 location.Orc, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
590 location.Hectometre, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
591 &validity, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
592 lnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
593 mwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
594 hnwl, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
595 fe30, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
596 fe100, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
597 dateInfo, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
598 source, |
2094
7a3378c14397
Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2093
diff
changeset
|
599 precision, |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
600 ).Scan(&id, &noGeom); err != nil { |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
601 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
602 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
603 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
604 if _, err := trackStmt.ExecContext( |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
605 ctx, importID, "waterway.waterway_profiles", id); err != nil { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
606 return nil, err |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
607 } |
2092
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
608 |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
609 if noGeom { |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
610 feedback.Warn( |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
611 "No profile geometry found for %s in line %d.", location, line) |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
612 } |
f70f67eaa7aa
Waterway profile import: log warnings if no profile geometry can be found for ISRS code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2091
diff
changeset
|
613 |
2079
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
614 ids = append(ids, id) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
615 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
616 if len(ids) == 0 { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
617 return nil, UnchangedError("No new entries in waterway profiles.") |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
618 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
619 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
620 feedback.Info("%d new entries in waterway profiles.", len(ids)) |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
621 |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
622 summary := struct { |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
623 IDs []int64 `json:"ids"` |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
624 }{ |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
625 IDs: ids, |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
626 } |
9318973487a1
Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2078
diff
changeset
|
627 return &summary, nil |
2072
b4d8d320feab
Waterway profile import: Added import stub and changed database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
628 } |