gemma: pkg/imports/wp.go annotate

annotate pkg/imports/wp.go @ 2143:f0641b5ad065

Waterway profile imports: Fixed typo in error message.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 07 Feb 2019 15:12:04 +0100
parents	a1f2cfa624cf
children	b868cb653c4d

rev	line source
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	17 "bufio"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	20 "encoding/csv"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	22 "errors"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	23 "fmt"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	24 "io"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "os"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	26 "path/filepath"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	27 "strconv"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	28 "strings"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	29 "time"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	31 "github.com/jackc/pgx/pgtype"
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	32
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 "gemma.intevation.de/gemma/pkg/common"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	34 "gemma.intevation.de/gemma/pkg/misc"
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	36 "gemma.intevation.de/gemma/pkg/wfs"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 )
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	39 // defaultPointToLinePrecision is the precision in meters
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	40 // to match from points to lines.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	41 const defaultPointToLinePrecision = 10
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	42
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 Dir string `json:"dir"`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	45 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	46 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	47 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	48 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	49 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	50 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	51 SortBy string `json:"sort-by"`
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	52 // Precsion of match points to line strings.
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	53 Precision *float64 `json:"precision,omitempty"`
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 func (wpJobCreator) Create(_ JobKind, data string) (Job, error) {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 wp := new(WaterwayProfiles)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 if err := common.FromJSONString(data, wp); err != nil {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 return nil, err
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 return wp, nil
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78 func (wpJobCreator) Depends() []string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	79 return []string{
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 "waterway_profiles",
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	82 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	83
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	84 const (
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	85 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	86 CREATE TEMP TABLE wp_geoms (
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	87 geom geography(linestring, 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	88 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	89
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	90 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	91 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	92
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	93 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	94
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	95 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	96 INSERT INTO wp_geoms (geom) VALUES (
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	97 ST_Transform(ST_GeomFromWKB($1, $2::int), 4326)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	98 )`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	99
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	100 insertWaterwayProfileSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	101 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	102 location,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	103 geom,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	104 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	105 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	106 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	107 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	108 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	109 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	110 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	111 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	112 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	113 ($1, $2, $3, $4, $5),
2093 a9d28c489761 Waterway profile import: Do all the calculations in geography domain. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2092 diff changeset	114 ( SELECT wp_geoms.geom
2127 a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	115 FROM wp_geoms, waterway.distance_marks_virtual AS dmv
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	116 WHERE dmv.location_code =
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	117 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	118 AND ST_DWithin(dmv.geom, wp_geoms.geom, $14::float)
a1f2cfa624cf Make distance filtering more consistent Tom Gottfried <tom@intevation.de> parents: 2126 diff changeset	119 ORDER BY ST_Distance(dmv.geom, wp_geoms.geom, true)
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	120 LIMIT 1
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	121 ),
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	122 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	123 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	124 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	125 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	126 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	127 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	128 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	129 $13
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	130 ) RETURNING id, geom is NULL`
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	131
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	132 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	133 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	134 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	135 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	136 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	137 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	138 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	139
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	140 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	141 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	142 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	143 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	144 ) error {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	145 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	146 return err
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	148
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	149 func (wp *WaterwayProfiles) CleanUp() error {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	150 return os.RemoveAll(wp.Dir)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	151 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	152
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	153 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	154 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	155 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	156 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	157 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	158 ) (interface{}, error) {
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	159 start := time.Now()
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	160
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	161 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	162 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	163 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	164 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	165 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	166
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	167 if err := wp.downloadGeometries(
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	168 ctx, importID, tx, start, feedback); err != nil {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	169 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	170 }
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	171
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	172 summary, err := wp.processCSV(
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	173 ctx, importID, tx, start, feedback)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	174 if err != nil {
2143 f0641b5ad065 Waterway profile imports: Fixed typo in error message. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2127 diff changeset	175 return nil, fmt.Errorf("error processing CSV: %v", err)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	176 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	177
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	178 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	179 return nil, fmt.Errorf(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	180 "Importing waterway profiles failed after %s: %v",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	181 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	182 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	183
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	184 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	185 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	186 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	187 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	188
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	189 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	190 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	191 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	192 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	193 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	194 feedback Feedback,
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	195 ) error {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	196 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	197
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	198 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	199 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	200 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	201 feedback.Error("Loading capabilities failed: %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	202 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	203 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	204
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	205 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	206 if ft == nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	207 return fmt.Errorf("Unknown feature type '%s'", wp.FeatureType)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	208 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	209
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	210 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	211
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	212 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	213 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	214 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	215 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	216 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	217
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	218 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	219 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	220 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	221
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	222 urls, err := wfs.GetFeaturesGET(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	223 caps, wp.FeatureType, "application/json", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	224 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	225 feedback.Error("Cannot create GetFeature URLs. %v", err)
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	226 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	227 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	228
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	229 if _, err := tx.ExecContext(ctx, createGeomTempTableSQL); err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	230 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	231 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	232
2126 d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	233 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	234 return err
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	235 }
d4a5e475f874 Actually create index on temp table Tom Gottfried <tom@intevation.de> parents: 2109 diff changeset	236
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	237 insertStmt, err := tx.PrepareContext(ctx, insertGeomTmpTableSQL)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	238 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	239 return err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	240 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	241 defer insertStmt.Close()
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	242
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	243 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	244 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	245 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	246 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	247 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	248
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2094 diff changeset	249 if err := wfs.DownloadURLs(urls, func(url string, r io.Reader) error {
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2094 diff changeset	250 feedback.Info("Get features from: '%s'", url)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	251 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	252 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	253 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	254 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	255 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	256 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	257 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	258 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	259 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	260 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	261 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	262
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	263 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	264 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	265 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	266 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	267
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	268 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	269 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	270 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	271 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	272 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	273
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	274 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	275 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	276 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	277 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	278 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	279 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	280 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	281 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	282 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	283 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	284 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	285 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	286 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	287 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	288 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	289 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	290 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	291 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	292 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	293 }); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	294 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	295 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	296
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	297 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	298 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	299 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	300
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	301 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	302 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	303 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	304
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	305 if features == 0 {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	306 return errors.New("No features found")
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	307 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	308 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	309 return err
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	310 }
2091 3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	311 return nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	312 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	313
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	314 func parseFloat64(s string) (sql.NullFloat64, error) {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	315 if s == "" {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	316 return sql.NullFloat64{}, nil
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	317 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	318 s = strings.Replace(s, ",", ".", -1)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	319 v, err := strconv.ParseFloat(s, 64)
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	320 if err != nil {
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	321 return sql.NullFloat64{}, err
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	322 }
3f32b18bbd4c Waterway profile import: Store temp geoms in WGS84, do buffering with four decimal places and fetch the lines closest to the dmv points. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2087 diff changeset	323 return sql.NullFloat64{Float64: v, Valid: true}, nil
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	324 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	325
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	326 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	327 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	328 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	329 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	330 start time.Time,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	331 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	332 ) (interface{}, error) {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	333 feedback.Info("Start processing CSV file.")
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	334
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	335 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	336 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	337 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	338 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	339 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	340
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	341 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	342 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	343 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	344
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	345 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	346 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	347 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	348 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	349
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	350 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	351 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	352 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	353 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	354 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	355 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	356 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	357 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	358 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	359 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	360 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	361 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	362
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	363 fields := []struct {
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	364 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	365 substr string
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	366 }{
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	367 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	368 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	369 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	370 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	371 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	372 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	373 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	374 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	375 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	376 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	377 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	378
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	379 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	380 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	381 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	382 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	383 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	384 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	385 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	386 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	387 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	388 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	389 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	390 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	391 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	392 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	393 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	394
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	395 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	396 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	397 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	398 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	399 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	400 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	401 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	402 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	403 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	404 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	405 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	406
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	407 var precision float64
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	408 if wp.Precision != nil {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	409 if precision = *wp.Precision; precision < 0 {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	410 precision = -precision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	411 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	412 } else {
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	413 precision = defaultPointToLinePrecision
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	414 }
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	415
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	416 feedback.Info(
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	417 "Matching points to lines with a precision of %.4fm.", precision)
7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	418
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	419 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	420
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	421 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	422 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	423 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	424 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	425 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	426
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	427 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	428 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	429 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	430 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	431 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	432
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	433 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	434
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	435 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	436 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	437
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	438 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	439 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	440 case err == io.EOF \|\| len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	441 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	442 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	443 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	444 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	445
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	446 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	447 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	448 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	449 "Invalid ISRS location code in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	450 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	451 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	452
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	453 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	454 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	455 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	456 "Invalid 'valid_from' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	457 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	458 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	459 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	460 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	461 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	462 "Invalid 'valid_to' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	463 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	464 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	465
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	466 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	467 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	468 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	469 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	470
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	471 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	472 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	473 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	474 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	475
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	476 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	477 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	478 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	479 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	480 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	481 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	482 }
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	483
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	484 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	485 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	486 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	487 "Invalid 'lnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	488 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	489 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	490 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	491 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	492 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	493 "Invalid 'mwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	494 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	495 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	496 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	497 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	498 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	499 "Invalid 'hnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	500 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	501 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	502 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	503 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	504 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	505 "Invalid 'fe30' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	506 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	507 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	508 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	509 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	510 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	511 "Invalid 'fe100' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	512 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	513 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	514
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	515 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	516
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	517 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	518 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	519 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	520 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	521 "Invalid 'date_info' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	522 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	523 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	524
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	525 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	526
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	527 var id int64
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	528 var noGeom bool
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	529
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	530 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	531 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	532 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	533 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	534 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	535 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	536 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	537 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	538 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	539 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	540 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	541 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	542 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	543 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	544 source,
2094 7a3378c14397 Waterway profiles import: Added a precision parameter in meters to find lines for points (default 10m). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2093 diff changeset	545 precision,
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	546 ).Scan(&id, &noGeom); err != nil {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	547 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	548 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	549
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	550 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	551 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	552 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	553 }
2092 f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	554
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	555 if noGeom {
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	556 feedback.Warn(
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	557 "No profile geometry found for %s in line %d.", location, line)
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	558 }
f70f67eaa7aa Waterway profile import: log warnings if no profile geometry can be found for ISRS code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2091 diff changeset	559
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	560 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	561 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	562 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	563 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	564 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	565
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	566 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	567
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	568 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	569 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	570 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	571 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	572 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	573 return &summary, nil
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	574 }

Mercurial > gemma

annotate pkg/imports/wp.go @ 2143:f0641b5ad065