gemma: pkg/imports/wp.go annotate

annotate pkg/imports/wp.go @ 2087:5d3d2e823314

Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Fri, 01 Feb 2019 10:56:29 +0100
parents	6096ec4951f8
children	3f32b18bbd4c

rev	line source
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	17 "bufio"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	20 "encoding/csv"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	22 "errors"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	23 "fmt"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	24 "io"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "os"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	26 "path/filepath"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	27 "strconv"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	28 "strings"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	29 "time"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/common"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	32 "gemma.intevation.de/gemma/pkg/misc"
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	33 "gemma.intevation.de/gemma/pkg/models"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	34 "gemma.intevation.de/gemma/pkg/wfs"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	35 "github.com/jackc/pgx/pgtype"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 Dir string `json:"dir"`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	40 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	41 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	42 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	43 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	44 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	45 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	46 SortBy string `json:"sort-by"`
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 func (wpJobCreator) Create(_ JobKind, data string) (Job, error) {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 wp := new(WaterwayProfiles)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 if err := common.FromJSONString(data, wp); err != nil {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 return nil, err
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 return wp, nil
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 func (wpJobCreator) Depends() []string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 return []string{
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 "waterway_profiles",
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	77 const (
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	78 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	79 CREATE TEMP TABLE wp_geoms (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	80 geom geometry(linestring, %d)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	81 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	82
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	83 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	84 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	85
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	86 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	87
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	88 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	89 INSERT INTO wp_geoms (geom) VALUES (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	90 ST_Transform(ST_GeomFromWKB($1, $2::int), %d)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	91 )`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	92
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	93 insertWaterwayProfileSQL = `
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	94 WITH point AS (
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	95 SELECT ST_Buffer(ST_Transform(geom::geometry, $14::int), 0.001) geom
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	96 FROM waterway.distance_marks_virtual
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	97 WHERE location_code =
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	98 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	99 )
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	100 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	101 location,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	102 geom,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	103 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	104 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	105 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	106 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	107 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	108 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	109 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	110 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	111 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	112 ($1, $2, $3, $4, $5),
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	113 ( SELECT ST_Transform(geom, 4326)::geography
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	114 FROM wp_geoms
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	115 WHERE geom && ( SELECT geom from point ) AND
2087 5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	116 ST_Intersects(geom, ( SELECT geom FROM point ))
5d3d2e823314 Waterway profiles: Do a little buffeing around the point to be more tolerant when finding the right line string. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2086 diff changeset	117 LIMIT 1
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	118 ),
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	119 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	120 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	121 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	122 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	123 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	124 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	125 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	126 $13
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	127 ) RETURNING id`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	128
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	129 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	130 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	131 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	132 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	133 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	134 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	135 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	136
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	137 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	138 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	139 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	140 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	141 ) error {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	142 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	143 return err
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	144 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	145
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	146 func (wp *WaterwayProfiles) CleanUp() error {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147 return os.RemoveAll(wp.Dir)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	148 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	149
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	150 func parseFloat64(s string) (sql.NullFloat64, error) {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	151 if s == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	152 return sql.NullFloat64{}, nil
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	153 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	154 s = strings.Replace(s, ",", ".", -1)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	155 v, err := strconv.ParseFloat(s, 64)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	156 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	157 return sql.NullFloat64{}, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	158 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	159 return sql.NullFloat64{Float64: v, Valid: true}, nil
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	160 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	161
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	162 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	163 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	164 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	165 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	166 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	167 ) (interface{}, error) {
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	168 start := time.Now()
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	169
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	170 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	171 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	172 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	173 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	174 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	175
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	176 epsg, err := wp.downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	177 ctx, importID, tx, start, feedback)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	178 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	179 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	180 }
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	181
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	182 summary, err := wp.processCSV(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	183 ctx, importID, tx, start, epsg, feedback)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	184 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	185 return nil, fmt.Errorf("error processing CVS: %v", err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	186 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	187
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	188 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	189 return nil, fmt.Errorf(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	190 "Importing waterway profiles failed after %s: %v",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	191 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	192 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	193
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	194 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	195 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	196 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	197 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	198
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	199 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	200 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	201 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	202 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	203 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	204 feedback Feedback,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	205 ) (int, error) {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	206 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	207
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	208 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	209 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	210 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	211 feedback.Error("Loading capabilities failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	212 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	213 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	214
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	215 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	216 if ft == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	217 return 0, fmt.Errorf("Unknown feature type '%s'", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	218 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	219
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	220 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	221
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	222 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	223 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	224 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	225 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	226 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	227
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	228 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	229 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	230 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	231
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	232 urls, err := wfs.GetFeaturesGET(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	233 caps, wp.FeatureType, "application/json", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	234 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	235 feedback.Error("Cannot create GetFeature URLs. %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	236 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	237 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	238
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	239 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	240 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	241 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	242 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	243 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	244
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	245 var insertStmt *sql.Stmt
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	246 defer func() {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	247 if insertStmt != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	248 insertStmt.Close()
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	249 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	250 }()
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	251
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	252 var usedEPGS int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	253
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	254 setup := func(epsg int) error {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	255 if insertStmt != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	256 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	257 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	258 feedback.Info("Using EPSG: %d", epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	259 usedEPGS = epsg
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	260
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	261 tblSQL := fmt.Sprintf(createGeomTempTableSQL, epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	262 if _, err := tx.ExecContext(ctx, tblSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	263 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	264 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	265 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	266 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	267 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	268 insertSQL := fmt.Sprintf(insertGeomTmpTableSQL, epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	269 var err error
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	270 insertStmt, err = tx.PrepareContext(ctx, insertSQL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	271 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	272 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	273
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	274 if err := wfs.DownloadURLs(urls, func(r io.Reader) error {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	275 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	276 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	277 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	278 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	279 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	280 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	281 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	282 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	283 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	284 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	285 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	286
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	287 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	288 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	289 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	290 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	291
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	292 if err := setup(epsg); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	293 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	294 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	295
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	296 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	297 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	298 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	299 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	300 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	301
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	302 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	303 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	304 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	305 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	306 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	307 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	308 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	309 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	310 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	311 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	312 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	313 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	314 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	315 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	316 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	317 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	318 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	319 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	320 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	321 }); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	322 feedback.Error("Downloading features failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	323 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	324 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	325
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	326 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	327 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	328 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	329
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	330 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	331 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	332 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	333
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	334 if features == 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	335 return 0, errors.New("No features found")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	336 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	337 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	338 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	339 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	340 return usedEPGS, nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	341 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	342
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	343 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	344 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	345 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	346 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	347 start time.Time,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	348 epsg int,
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	349 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	350 ) (interface{}, error) {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	351 feedback.Info("Start processing CSV file.")
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	352
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	353 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	354 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	355 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	356 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	357 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	358
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	359 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	360 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	361 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	362
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	363 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	364 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	365 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	366 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	367
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	368 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	369 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	370 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	371 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	372 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	373 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	374 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	375 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	376 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	377 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	378 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	379 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	380
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	381 fields := []struct {
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	382 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	383 substr string
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	384 }{
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	385 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	386 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	387 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	388 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	389 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	390 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	391 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	392 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	393 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	394 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	395 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	396
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	397 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	398 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	399 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	400 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	401 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	402 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	403 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	404 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	405 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	406 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	407 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	408 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	409 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	410 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	411 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	412
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	413 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	414 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	415 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	416 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	417 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	418 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	419 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	420 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	421 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	422 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	423 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	424
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	425 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	426
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	427 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	428 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	429 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	430 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	431 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	432
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	433 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	434 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	435 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	436 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	437 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	438
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	439 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	440
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	441 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	442 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	443
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	444 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	445 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	446 case err == io.EOF \|\| len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	447 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	448 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	449 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	450 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	451
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	452 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	453 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	454 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	455 "Invalid ISRS location code in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	456 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	457 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	458
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	459 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	460 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	461 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	462 "Invalid 'valid_from' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	463 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	464 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	465 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	466 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	467 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	468 "Invalid 'valid_to' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	469 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	470 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	471
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	472 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	473 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	474 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	475 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	476
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	477 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	478 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	479 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	480 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	481
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	482 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	483 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	484 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	485 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	486 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	487 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	488 }
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	489
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	490 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	491 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	492 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	493 "Invalid 'lnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	494 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	495 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	496 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	497 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	498 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	499 "Invalid 'mwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	500 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	501 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	502 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	503 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	504 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	505 "Invalid 'hnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	506 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	507 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	508 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	509 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	510 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	511 "Invalid 'fe30' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	512 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	513 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	514 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	515 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	516 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	517 "Invalid 'fe100' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	518 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	519 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	520
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	521 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	522
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	523 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	524 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	525 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	526 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	527 "Invalid 'date_info' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	528 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	529 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	530
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	531 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	532
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	533 var id int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	534 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	535 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	536 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	537 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	538 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	539 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	540 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	541 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	542 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	543 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	544 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	545 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	546 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	547 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	548 source,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	549 epsg,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	550 ).Scan(&id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	551 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	552 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	553
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	554 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	555 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	556 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	557 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	558 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	559 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	560 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	561 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	562 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	563
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	564 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	565
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	566 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	567 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	568 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	569 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	570 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	571 return &summary, nil
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	572 }

Mercurial > gemma

annotate pkg/imports/wp.go @ 2087:5d3d2e823314