gemma: pkg/imports/wp.go annotate

annotate pkg/imports/wp.go @ 2187:7c83b5277c1c

Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Mon, 11 Feb 2019 18:35:01 +0100
parents	b27edd1f2353
children	02505fcff63c

rev	line source
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	17 "bufio"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	20 "encoding/csv"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	22 "errors"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	23 "fmt"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	24 "io"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "os"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	26 "path/filepath"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	27 "strconv"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	28 "strings"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	29 "time"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	31 "github.com/jackc/pgx/pgtype"
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	32
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	33 "gemma.intevation.de/gemma/pkg/misc"
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	34 "gemma.intevation.de/gemma/pkg/models"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	35 "gemma.intevation.de/gemma/pkg/wfs"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	38 // defaultPointToLinePrecision is the precision in meters
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	39 // to match from points to lines.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	40 const defaultPointToLinePrecision = 10
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	41
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 Dir string `json:"dir"`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	44 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	45 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	46 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	47 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	48 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	49 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	50 SortBy string `json:"sort-by"`
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	51 // Precsion of match points to line strings.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	52 Precision *float64 `json:"precision,omitempty"`
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2183 diff changeset	63 func (wpJobCreator) Create() Job { return new(WaterwayProfiles) }
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 func (wpJobCreator) Depends() []string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 return []string{
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 "waterway_profiles",
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	77 const (
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	78 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	79 CREATE TEMP TABLE wp_geoms (
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	80 geom geography(linestring, 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	81 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	82
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	83 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	84 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	85
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	86 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	87
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	88 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	89 INSERT INTO wp_geoms (geom) VALUES (
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	90 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	91 )`
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	92 hasDistanceMarkSQL = `
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	93 SELECT true FROM waterway.distance_marks_virtual
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	94 WHERE location_code =
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	95 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	96 LIMIT 1`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	97
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	98 insertWaterwayProfileSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	99 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	100 location,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	101 geom,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	102 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	103 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	104 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	105 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	106 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	107 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	108 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	109 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	110 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	111 ($1, $2, $3, $4, $5),
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	112 ( SELECT wp_geoms.geom
2127 a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	113 FROM wp_geoms, waterway.distance_marks_virtual AS dmv
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	114 WHERE dmv.location_code =
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	115 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	116 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	117 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true)
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	118 LIMIT 1
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	119 ),
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	120 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	121 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	122 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	123 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	124 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	125 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	126 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	127 $13
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	128 ) RETURNING id, geom is NULL`
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	129
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	130 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	131 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	132 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	133 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	134 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	135 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	136 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	137
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	138 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	139 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	140 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	141 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	142 ) error {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	143 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	144 return err
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	145 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	146
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147 func (wp *WaterwayProfiles) CleanUp() error {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	148 return os.RemoveAll(wp.Dir)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	149 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	150
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	151 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	152 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	153 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	154 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	155 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	156 ) (interface{}, error) {
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	157 start := time.Now()
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	158
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	159 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	160 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	161 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	162 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	163 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	164
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	165 if err := wp.downloadGeometries(
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	166 ctx, importID, tx, start, feedback); err != nil {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	167 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	168 }
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	169
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	170 summary, err := wp.processCSV(
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	171 ctx, importID, tx, start, feedback)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	172 if err != nil {
2143 f0641b5ad065 Waterway profile imports: Fixed typo in error message. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2127 diff changeset	173 return nil, fmt.Errorf("error processing CSV: %v", err)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	174 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	175
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	176 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	177 return nil, fmt.Errorf(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	178 "Importing waterway profiles failed after %s: %v",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	179 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	180 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	181
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	182 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	183 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	184 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	185 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	186
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	187 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	188 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	189 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	190 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	191 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	192 feedback Feedback,
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	193 ) error {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	194 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	195
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	196 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	197 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	198 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	199 feedback.Error("Loading capabilities failed: %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	200 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	201 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	202
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	203 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	204 if ft == nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	205 return fmt.Errorf("Unknown feature type '%s'", wp.FeatureType)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	206 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	207
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	208 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	209
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	210 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	211 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	212 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	213 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	214 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	215
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	216 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	217 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	218 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	219
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	220 urls, err := wfs.GetFeaturesGET(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	221 caps, wp.FeatureType, "application/json", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	222 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	223 feedback.Error("Cannot create GetFeature URLs. %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	224 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	225 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	226
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	227 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	228 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	229 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	230
2126 d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	231 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	232 return err
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	233 }
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	234
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	235 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	236 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	237 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	238 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	239 defer insertStmt.Close()
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	240
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	241 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	242 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	243 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	244 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	245 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	246
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2094 diff changeset	247 if err := wfs.DownloadURLs(urls, func(url string, r io.Reader) error {
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2094 diff changeset	248 feedback.Info("Get features from: '%s'", url)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	249 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	250 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	251 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	252 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	253 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	254 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	255 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	256 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	257 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	258 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	259 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	260
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	261 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	262 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	263 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	264 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	265
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	266 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	267 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	268 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	269 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	270 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	271
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	272 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	273 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	274 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	275 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	276 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	277 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	278 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	279 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	280 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	281 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	282 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	283 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	284 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	285 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	286 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	287 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	288 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	289 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	290 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	291 }); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	292 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	293 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	294
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	295 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	296 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	297 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	298
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	299 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	300 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	301 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	302
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	303 if features == 0 {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	304 return errors.New("No features found")
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	305 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	306 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	307 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	308 }
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	309 return nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	310 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	311
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	312 func parseFloat64(s string) (sql.NullFloat64, error) {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	313 if s == "" {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	314 return sql.NullFloat64{}, nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	315 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	316 s = strings.Replace(s, ",", ".", -1)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	317 v, err := strconv.ParseFloat(s, 64)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	318 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	319 return sql.NullFloat64{}, err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	320 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	321 return sql.NullFloat64{Float64: v, Valid: true}, nil
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	322 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	323
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	324 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	325 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	326 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	327 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	328 start time.Time,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	329 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	330 ) (interface{}, error) {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	331 feedback.Info("Start processing CSV file.")
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	332
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	333 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	334 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	335 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	336 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	337 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	338
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	339 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	340 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	341 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	342
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	343 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	344 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	345 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	346 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	347
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	348 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	349 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	350 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	351 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	352 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	353 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	354 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	355 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	356 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	357 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	358 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	359 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	360
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	361 fields := []struct {
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	362 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	363 substr string
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	364 }{
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	365 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	366 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	367 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	368 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	369 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	370 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	371 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	372 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	373 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	374 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	375 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	376
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	377 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	378 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	379 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	380 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	381 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	382 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	383 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	384 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	385 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	386 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	387 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	388 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	389 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	390 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	391 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	392
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	393 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	394 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	395 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	396 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	397 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	398 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	399 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	400 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	401 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	402 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	403 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	404
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	405 var precision float64
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	406 if wp.Precision != nil {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	407 if precision = *wp.Precision; precision < 0 {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	408 precision = -precision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	409 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	410 } else {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	411 precision = defaultPointToLinePrecision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	412 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	413
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	414 feedback.Info(
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	415 "Matching points to lines with a precision of %.4fm.", precision)
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	416
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	417 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	418
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	419 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	420 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	421 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	422 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	423 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	424
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	425 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	426 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	427 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	428 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	429 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	430
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	431 hasDistanceMarkStmt, err := tx.PrepareContext(ctx, hasDistanceMarkSQL)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	432 if err != nil {
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	433 return nil, err
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	434 }
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	435 defer hasDistanceMarkStmt.Close()
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	436
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	437 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	438
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	439 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	440 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	441
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	442 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	443 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	444 case err == io.EOF \|\| len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	445 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	446 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	447 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	448 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	449
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	450 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	451 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	452 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	453 "Invalid ISRS location code in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	454 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	455 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	456
2183 b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	457 var dummy bool
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	458 err = hasDistanceMarkStmt.QueryRowContext(
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	459 ctx,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	460 location.CountryCode,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	461 location.LoCode,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	462 location.FairwaySection,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	463 location.Orc,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	464 location.Hectometre,
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	465 ).Scan(&dummy)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	466
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	467 switch {
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	468 case err == sql.ErrNoRows:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	469 feedback.Warn("No virtual distance mark found for %s.", location)
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	470 continue lines
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	471 case err != nil:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 return nil, err
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 case !dummy:
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474 return nil, errors.New("unexpected result form database")
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 }
b27edd1f2353 Waterway profiles import: Issue warnings for lines in CSV file which ISRS codes which are not in the virtual distance marks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	477 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	478 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	479 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	480 "Invalid 'valid_from' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	481 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	482 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	483 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	484 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	485 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	486 "Invalid 'valid_to' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	487 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	488 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	489
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	490 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	491 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	492 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	493 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	494
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	495 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	496 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	497 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	498 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	499
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	500 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	501 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	502 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	503 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	504 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	505 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	506 }
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	507
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	508 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	509 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	510 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	511 "Invalid 'lnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	512 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	513 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	514 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	515 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	516 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	517 "Invalid 'mwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	518 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	519 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	520 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	521 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	522 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	523 "Invalid 'hnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	524 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	525 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	526 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	527 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	528 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	529 "Invalid 'fe30' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	530 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	531 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	532 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	533 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	534 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	535 "Invalid 'fe100' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	536 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	537 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	538
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	539 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	540
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	541 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	542 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	543 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	544 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	545 "Invalid 'date_info' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	546 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	547 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	548
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	549 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	550
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	551 var id int64
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	552 var noGeom bool
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	553
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	554 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	555 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	556 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	557 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	558 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	559 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	560 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	561 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	562 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	563 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	564 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	565 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	566 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	567 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	568 source,
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	569 precision,
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	570 ).Scan(&id, &noGeom); err != nil {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	571 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	572 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	573
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	574 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	575 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	576 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	577 }
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	578
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	579 if noGeom {
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	580 feedback.Warn(
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	581 "No profile geometry found for %s in line %d.", location, line)
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	582 }
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	583
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	584 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	585 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	586 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	587 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	588 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	589
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	590 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	591
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	592 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	593 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	594 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	595 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	596 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	597 return &summary, nil
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	598 }

Mercurial > gemma

annotate pkg/imports/wp.go @ 2187:7c83b5277c1c