gemma: pkg/imports/wp.go annotate

annotate pkg/imports/wp.go @ 4182:49012340336c

Made 'golint' finally happy with imports package.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Tue, 06 Aug 2019 11:13:28 +0200
parents	8b75ac5e243e
children	ca6a5f722471

rev	line source
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	17 "bufio"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	20 "encoding/csv"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	22 "errors"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	23 "fmt"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	24 "io"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "os"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	26 "path/filepath"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	27 "strconv"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	28 "strings"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	29 "time"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	31 "github.com/jackc/pgx/pgtype"
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	32
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	33 "gemma.intevation.de/gemma/pkg/common"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	34 "gemma.intevation.de/gemma/pkg/models"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	35 "gemma.intevation.de/gemma/pkg/wfs"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	38 // defaultPointToLinePrecision is the precision in meters
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	39 // to match from points to lines.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	40 const defaultPointToLinePrecision = 10
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	41
4182 49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	42 // WaterwayProfiles is a Job to import waterway profiles
49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	43 // from a given WFS service plus some uploaded CSV file
49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	44 // and stores them into the database.
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 Dir string `json:"dir"`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	47 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	48 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	49 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	50 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	51 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	52 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	53 SortBy string `json:"sort-by"`
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	54 // Precsion of match points to line strings.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	55 Precision *float64 `json:"precision,omitempty"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	56 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	57 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	58 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	59 Password string `json:"password,omitempty"`
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61
4182 49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	62 // WPJobKind is the unique name of this import job type.
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2183 diff changeset	71 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) }
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	79 func (wpJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	80 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	81 {"waterway_profiles"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	82 {"distance_marks_virtual"},
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	83 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	86 const (
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	87 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	88 CREATE TEMP TABLE wp_geoms (
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	89 geom geography(linestring, 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	90 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	91
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	92 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	93 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	94
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	95 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	96
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	97 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	98 INSERT INTO wp_geoms (geom) VALUES (
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	99 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	100 )`
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	101 hasDistanceMarkSQL = `
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	102 SELECT true FROM waterway.distance_marks_virtual
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	103 WHERE location_code =
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	104 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	105 LIMIT 1`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	106
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	107 insertWaterwayProfileSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	108 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	109 location,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	110 geom,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	111 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	112 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	113 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	114 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	115 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	116 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	117 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	118 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	119 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	120 ($1, $2, $3, $4, $5),
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	121 ( SELECT wp_geoms.geom
2127 a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	122 FROM wp_geoms, waterway.distance_marks_virtual AS dmv
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	123 WHERE dmv.location_code =
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	124 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	125 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	126 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true)
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	127 LIMIT 1
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	128 ),
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	129 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	130 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	131 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	132 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	133 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	134 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	135 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	136 $13
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	137 ) RETURNING id, geom is NULL`
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	138
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	139 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	140 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	141 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	142 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	143 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	144 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	145 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	146
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	148 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	149 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	150 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	151 ) error {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	152 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	153 return err
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	154 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	155
4182 49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	156 // CleanUp deletes temporary files from the filesystem.
49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	157 func (wp *WaterwayProfiles) CleanUp() error { return os.RemoveAll(wp.Dir) }
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	158
4182 49012340336c Made 'golint' finally happy with imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	159 // Do performs the actual import.
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	160 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	161 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	162 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	163 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	164 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	165 ) (interface{}, error) {
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	166 start := time.Now()
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	167
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	168 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	169 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	170 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	171 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	172 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	173
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	174 if err := wp.downloadGeometries(
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	175 ctx, importID, tx, start, feedback); err != nil {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	176 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	177 }
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	178
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	179 summary, err := wp.processCSV(
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	180 ctx, importID, tx, start, feedback)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	181 if err != nil {
2143 f0641b5ad065 Waterway profile imports: Fixed typo in error message. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2127 diff changeset	182 return nil, fmt.Errorf("error processing CSV: %v", err)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	183 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	184
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	185 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	186 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	187 "importing waterway profiles failed after %s: %v",
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	188 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	189 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	190
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	191 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	192 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	193 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	194 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	195
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	196 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	197 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	198 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	199 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	200 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	201 feedback Feedback,
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	202 ) error {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	203 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	204
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	205 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	206 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	207 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	208 feedback.Error("Loading capabilities failed: %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	209 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	210 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	211
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	212 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	213 if ft == nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	214 return fmt.Errorf("unknown feature type '%s'", wp.FeatureType)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	215 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	216
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	217 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	218
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	219 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	220 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	221 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	222 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	223 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	224
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	225 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	226 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	227 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	228
2721 56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2719 diff changeset	229 dl, err := wfs.GetFeatures(caps, wp.FeatureType, wp.SortBy)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	230 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	231 feedback.Error("Cannot create GetFeature URLs. %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	232 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	233 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	234
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	235 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	236 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	237 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	238
2126 d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	239 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	240 return err
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	241 }
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	242
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	243 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	244 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	245 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	246 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	247 defer insertStmt.Close()
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	248
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	249 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	250 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	251 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	252 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	253 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	254
2729 7cb027be277d WFS downloader: Really use username and password from parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2723 diff changeset	255 if err := dl.Download(wp.User, wp.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2094 diff changeset	256 feedback.Info("Get features from: '%s'", url)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	257 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	258 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	259 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	260 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	261 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	262 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	263 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	264 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	265 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	266 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	267 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	268
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	269 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	270 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	271 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	272 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	273
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	274 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	275 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	276 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	277 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	278 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	279
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	280 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	281 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	282 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	283 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	284 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	285 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	286 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	287 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	288 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	289 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	290 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	291 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	292 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	293 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	294 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	295 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	296 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	297 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	298 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	299 }); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	300 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	301 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	302
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	303 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	304 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	305 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	306
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	307 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	308 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	309 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	310
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	311 if features == 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	312 return errors.New("no features found")
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	313 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	314 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	315 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	316 }
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	317 return nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	318 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	319
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	320 func parseFloat64(s string) (sql.NullFloat64, error) {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	321 if s == "" {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	322 return sql.NullFloat64{}, nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	323 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	324 s = strings.Replace(s, ",", ".", -1)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	325 v, err := strconv.ParseFloat(s, 64)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	326 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	327 return sql.NullFloat64{}, err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	328 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	329 return sql.NullFloat64{Float64: v, Valid: true}, nil
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	330 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	331
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	332 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	333 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	334 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	335 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	336 start time.Time,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	337 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	338 ) (interface{}, error) {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	339 feedback.Info("Start processing CSV file.")
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	340
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	341 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	342 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	343 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	344 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	345 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	346
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	347 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	348 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	349 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	350
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	351 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	352 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	353 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	354 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	355
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	356 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	357 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	358 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	359 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	360 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	361 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	362 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	363 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	364 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	365 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	366 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	367 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	368
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	369 fields := []struct {
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	370 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	371 substr string
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	372 }{
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	373 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	374 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	375 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	376 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	377 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	378 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	379 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	380 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	381 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	382 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	383 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	384
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	385 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	386 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	387 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	388 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	389 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	390 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	391 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	392 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	393 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	394 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	395 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	396 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	397 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	398 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	399 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	400
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	401 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	402 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	403 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	404 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	405 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	406 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	407 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	408 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	409 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	410 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	411 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	412
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	413 var precision float64
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	414 if wp.Precision != nil {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	415 if precision = *wp.Precision; precision < 0 {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	416 precision = -precision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	417 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	418 } else {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	419 precision = defaultPointToLinePrecision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	420 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	421
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	422 feedback.Info(
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	423 "Matching points to lines with a precision of %.4fm.", precision)
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	424
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	425 parseDate := common.TimeParser([]string{"02.01.2006"}).Parse
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	426
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	427 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	428 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	429 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	430 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	431 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	432
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	433 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	434 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	435 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	436 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	437 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	438
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	439 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	440 if err != nil {
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	441 return nil, err
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	442 }
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	443 defer hasDistanceMarkStmt.Close()
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	444
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	445 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	446
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	447 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	448 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	449
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	450 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	451 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	452 case err == io.EOF \|\| len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	453 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	454 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	455 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	456 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	457
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	458 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	459 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	460 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	461 "invalid ISRS location code in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	462 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	463 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	464
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	465 var dummy bool
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	466 err = hasDistanceMarkStmt.QueryRowContext(
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	467 ctx,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	468 location.CountryCode,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	469 location.LoCode,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	470 location.FairwaySection,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	471 location.Orc,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 location.Hectometre,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 ).Scan(&dummy)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 switch {
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476 case err == sql.ErrNoRows:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	477 feedback.Warn("No virtual distance mark found for %s.", location)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	478 continue lines
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	479 case err != nil:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	480 return nil, err
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 case !dummy:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	482 return nil, errors.New("unexpected result form database")
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	483 }
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	484
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	485 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	486 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	487 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	488 "invalid 'valid_from' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	489 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	490 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	491 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	492 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	493 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	494 "invalid 'valid_to' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	495 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	496 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	497
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	498 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	499 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	500 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	501 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	502
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	503 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	504 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	505 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	506 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	507
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	508 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	509 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	510 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	511 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	512 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	513 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	514 }
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	515
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	516 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	517 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	518 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	519 "invalid 'lnwl' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	520 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	521 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	522 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	523 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	524 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	525 "invalid 'mwl' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	526 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	527 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	528 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	529 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	530 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	531 "invalid 'hnwl' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	532 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	533 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	534 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	535 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	536 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	537 "invalid 'fe30' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	538 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	539 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	540 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	541 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	542 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	543 "invalid 'fe100' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	544 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	545 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	546
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	547 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	548
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	549 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	550 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	551 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	552 return nil, fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4075 diff changeset	553 "invalid 'date_info' value in line %d: %v",
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	554 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	555 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	556
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	557 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	558
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	559 var id int64
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	560 var noGeom bool
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	561
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	562 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	563 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	564 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	565 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	566 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	567 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	568 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	569 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	570 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	571 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	572 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	573 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	574 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	575 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	576 source,
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	577 precision,
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	578 ).Scan(&id, &noGeom); err != nil {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	579 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	580 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	581
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	582 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	583 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	584 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	585 }
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	586
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	587 if noGeom {
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	588 feedback.Warn(
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	589 "No profile geometry found for %s in line %d.", location, line)
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	590 }
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	591
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	592 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	593 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	594 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	595 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	596 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	597
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	598 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	599
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	600 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	601 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	602 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	603 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	604 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	605 return &summary, nil
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	606 }

Mercurial > gemma

annotate pkg/imports/wp.go @ 4182:49012340336c