gemma: pkg/imports/wp.go annotate

annotate pkg/imports/wp.go @ 2086:6096ec4951f8

Waterway profiles imports: Download the geometries from WPS and join them with the CSV data.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 31 Jan 2019 18:42:57 +0100
parents	ddbac0f22ffb
children	5d3d2e823314

rev	line source
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	17 "bufio"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	20 "encoding/csv"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	21 "encoding/json"
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	22 "errors"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	23 "fmt"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	24 "io"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "os"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	26 "path/filepath"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	27 "strconv"
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	28 "strings"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	29 "time"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/common"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	32 "gemma.intevation.de/gemma/pkg/misc"
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	33 "gemma.intevation.de/gemma/pkg/models"
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	34 "gemma.intevation.de/gemma/pkg/wfs"
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	35 "github.com/jackc/pgx/pgtype"
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 type WaterwayProfiles struct {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 Dir string `json:"dir"`
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	40 // URL the GetCapabilities URL of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	41 URL string `json:"url"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	42 // FeatureType selects the feature type of the WFS service.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	43 FeatureType string `json:"feature-type"`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	44 // SortBy works around misconfigured services to
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	45 // establish a sort order to get the features.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	46 SortBy string `json:"sort-by"`
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 const WPJobKind JobKind = "wp"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 type wpJobCreator struct{}
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 func init() {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 RegisterJobCreator(WPJobKind, wpJobCreator{})
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 func (wpJobCreator) Create(_ JobKind, data string) (Job, error) {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 wp := new(WaterwayProfiles)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 if err := common.FromJSONString(data, wp); err != nil {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 return nil, err
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 return wp, nil
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 func (wpJobCreator) AutoAccept() bool { return false }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 func (wpJobCreator) Description() string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 return "waterway profiles"
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 func (wpJobCreator) Depends() []string {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 return []string{
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 "waterway_profiles",
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	77 const (
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	78 createGeomTempTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	79 CREATE TEMP TABLE wp_geoms (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	80 geom geometry(linestring, %d)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	81 ) ON COMMIT DROP`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	82
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	83 createTempIndexSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	84 CREATE INDEX ON wp_geoms USING GIST(geom)`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	85
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	86 analyzeTempTableSQL = `ANALYZE wp_geoms`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	87
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	88 insertGeomTmpTableSQL = `
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	89 INSERT INTO wp_geoms (geom) VALUES (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	90 ST_Transform(ST_GeomFromWKB($1, $2::int), %d)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	91 )`
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	92
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	93 insertWaterwayProfileSQL = `
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	94 WITH point AS (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	95 SELECT ST_Transform(geom::geometry, $14::int) geom
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	96 FROM waterway.distance_marks_virtual
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	97 WHERE location_code =
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	98 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	99 )
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	100 INSERT INTO waterway.waterway_profiles (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	101 location,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	102 geom,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	103 validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	104 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	105 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	106 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	107 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	108 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	109 date_info,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	110 source_organization
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	111 ) VALUES (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	112 ($1, $2, $3, $4, $5),
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	113 ( SELECT ST_Transform(geom, 4326)::geography
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	114 FROM wp_geoms
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	115 WHERE geom && ( SELECT geom from point ) AND
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	116 ST_Intersects(geom, ( SELECT geom FROM point ))
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	117 ),
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	118 $6,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	119 $7,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	120 $8,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	121 $9,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	122 $10,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	123 $11,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	124 $12,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	125 $13
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	126 ) RETURNING id`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	127
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	128 wpStageDoneSQL = `
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	129 UPDATE waterway.waterway_profiles SET staging_done = true
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	130 WHERE id IN (
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	131 SELECT key FROM import.track_imports
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	132 WHERE import_id = $1 AND
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	133 relation = 'waterway.waterway_profiles'::regclass)`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	134 )
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	135
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	136 func (wpJobCreator) StageDone(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	137 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	138 tx *sql.Tx,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	139 id int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	140 ) error {
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	141 _, err := tx.ExecContext(ctx, wpStageDoneSQL, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	142 return err
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	143 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	144
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	145 func (wp *WaterwayProfiles) CleanUp() error {
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	146 return os.RemoveAll(wp.Dir)
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147 }
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	148
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	149 func parseFloat64(s string) (sql.NullFloat64, error) {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	150 if s == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	151 return sql.NullFloat64{}, nil
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	152 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	153 s = strings.Replace(s, ",", ".", -1)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	154 v, err := strconv.ParseFloat(s, 64)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	155 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	156 return sql.NullFloat64{}, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	157 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	158 return sql.NullFloat64{Float64: v, Valid: true}, nil
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	159 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	160
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	161 func (wp *WaterwayProfiles) Do(
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	162 ctx context.Context,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	163 importID int64,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	164 conn *sql.Conn,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	165 feedback Feedback,
b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	166 ) (interface{}, error) {
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	167 start := time.Now()
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	168
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	169 tx, err := conn.BeginTx(ctx, nil)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	170 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	171 return nil, err
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	172 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	173 defer tx.Rollback()
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	174
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	175 epsg, err := wp.downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	176 ctx, importID, tx, start, feedback)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	177 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	178 return nil, fmt.Errorf("error downloading geometries: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	179 }
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	180
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	181 summary, err := wp.processCSV(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	182 ctx, importID, tx, start, epsg, feedback)
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	183 if err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	184 return nil, fmt.Errorf("error processing CVS: %v", err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	185 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	186
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	187 if err := tx.Commit(); err != nil {
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	188 return nil, fmt.Errorf(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	189 "Importing waterway profiles failed after %s: %v",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	190 time.Since(start), err)
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	191 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	192
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	193 feedback.Info("Importing waterway profiles took %s",
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	194 time.Since(start))
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	195 return summary, nil
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	196 }
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	197
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	198 func (wp *WaterwayProfiles) downloadGeometries(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	199 ctx context.Context,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	200 importID int64,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	201 tx *sql.Tx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	202 start time.Time,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	203 feedback Feedback,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	204 ) (int, error) {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	205 feedback.Info("Start downloading geometries from WFS.")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	206
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	207 feedback.Info("Loading capabilities from %s", wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	208 caps, err := wfs.GetCapabilities(wp.URL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	209 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	210 feedback.Error("Loading capabilities failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	211 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	212 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	213
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	214 ft := caps.FindFeatureType(wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	215 if ft == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	216 return 0, fmt.Errorf("Unknown feature type '%s'", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	217 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	218
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	219 feedback.Info("Found feature type '%s", wp.FeatureType)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	220
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	221 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	222 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	223 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	224 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	225 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	226
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	227 if wp.SortBy != "" {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	228 feedback.Info("Features will be sorted by '%s'", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	229 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	230
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	231 urls, err := wfs.GetFeaturesGET(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	232 caps, wp.FeatureType, "application/json", wp.SortBy)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	233 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	234 feedback.Error("Cannot create GetFeature URLs. %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	235 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	236 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	237
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	238 var (
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	239 unsupported = stringCounter{}
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	240 missingProperties int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	241 features int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	242 )
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	243
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	244 var insertStmt *sql.Stmt
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	245 defer func() {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	246 if insertStmt != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	247 insertStmt.Close()
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	248 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	249 }()
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	250
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	251 var usedEPGS int
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	252
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	253 setup := func(epsg int) error {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	254 if insertStmt != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	255 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	256 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	257 feedback.Info("Using EPSG: %d", epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	258 usedEPGS = epsg
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	259
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	260 tblSQL := fmt.Sprintf(createGeomTempTableSQL, epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	261 if _, err := tx.ExecContext(ctx, tblSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	262 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	263 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	264 if _, err := tx.ExecContext(ctx, createTempIndexSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	265 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	266 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	267 insertSQL := fmt.Sprintf(insertGeomTmpTableSQL, epsg)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	268 var err error
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	269 insertStmt, err = tx.PrepareContext(ctx, insertSQL)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	270 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	271 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	272
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	273 if err := wfs.DownloadURLs(urls, func(r io.Reader) error {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	274 rfc, err := wfs.ParseRawFeatureCollection(r)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	275 if err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	276 return fmt.Errorf("parsing GetFeature document failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	277 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	278 if rfc.CRS != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	279 crsName := rfc.CRS.Properties.Name
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	280 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	281 feedback.Error("Unsupported CRS: %d", crsName)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	282 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	283 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	284 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	285
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	286 // No features -> ignore.
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	287 if rfc.Features == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	288 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	289 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	290
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	291 if err := setup(epsg); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	292 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	293 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	294
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	295 for _, feature := range rfc.Features {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	296 if feature.Geometry.Coordinates == nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	297 missingProperties++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	298 continue
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	299 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	300
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	301 switch feature.Geometry.Type {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	302 case "LineString":
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	303 var l lineSlice
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	304 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	305 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	306 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	307 if _, err := insertStmt.ExecContext(
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	308 ctx,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	309 l.asWKB(),
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	310 epsg,
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	311 ); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	312 return err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	313 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	314 features++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	315 default:
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	316 unsupported[feature.Geometry.Type]++
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	317 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	318 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	319 return nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	320 }); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	321 feedback.Error("Downloading features failed: %v", err)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	322 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	323 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	324
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	325 if missingProperties > 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	326 feedback.Warn("Missing properties: %d", missingProperties)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	327 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	328
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	329 if len(unsupported) != 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	330 feedback.Warn("Unsupported types found: %s", unsupported)
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	331 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	332
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	333 if features == 0 {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	334 return 0, errors.New("No features found")
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	335 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	336 if _, err := tx.ExecContext(ctx, analyzeTempTableSQL); err != nil {
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	337 return 0, err
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	338 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	339 return usedEPGS, nil
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	340 }
6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	341
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	342 func (wp *WaterwayProfiles) processCSV(
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	343 ctx context.Context,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	344 importID int64,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	345 tx *sql.Tx,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	346 start time.Time,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	347 epsg int,
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	348 feedback Feedback,
ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	349 ) (interface{}, error) {
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	350 feedback.Info("Start processing CSV file.")
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	351
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	352 f, err := os.Open(filepath.Join(wp.Dir, "wp.csv"))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	353 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	354 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	355 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	356 defer f.Close()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	357
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	358 r := csv.NewReader(bufio.NewReader(f))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	359 r.Comma = ';'
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	360 r.ReuseRecord = true
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	361
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	362 headers, err := r.Read()
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	363 if err != nil {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	364 return nil, err
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	365 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	366
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	367 var (
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	368 locationIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	369 validFromIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	370 validToIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	371 lnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	372 mwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	373 hnwlIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	374 fe30Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	375 fe100Idx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	376 dateInfoIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	377 sourceIdx = -1
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	378 )
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	379
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	380 fields := []struct {
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	381 idx *int
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	382 substr string
2084 ddbac0f22ffb Waterway profiles import: Restructured code a bit in preparation of downloading Profile geometries from WFS first (needs to be done). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2079 diff changeset	383 }{
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	384 {&locationIdx, "location"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	385 {&validFromIdx, "valid_from"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	386 {&validToIdx, "valid_to"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	387 {&lnwlIdx, "lnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	388 {&mwlIdx, "mwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	389 {&hnwlIdx, "hnwl"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	390 {&fe30Idx, "fe30"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	391 {&fe100Idx, "fe100"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	392 {&dateInfoIdx, "date_info"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	393 {&sourceIdx, "source"},
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	394 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	395
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	396 nextHeader:
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	397 for i, h := range headers {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	398 h = strings.ToLower(h)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	399 for j := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	400 if strings.Contains(h, fields[j].substr) {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	401 if *fields[j].idx != -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	402 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	403 "CSV has more than one column with name containing '%s'",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	404 fields[j].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	405 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	406 *fields[j].idx = i
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	407 continue nextHeader
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	408 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	409 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	410 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	411
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	412 var missing []string
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	413 for i := range fields {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	414 if *fields[i].idx == -1 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	415 missing = append(missing, fields[i].substr)
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	416 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	417 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	418 if len(missing) > 0 {
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	419 return nil, fmt.Errorf(
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	420 "CSV is missing columns: %s",
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	421 strings.Join(missing, ", "))
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	422 }
dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	423
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	424 parseDate := misc.TimeGuesser([]string{"02.01.2006"}).Guess
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	425
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	426 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayProfileSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	427 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	428 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	429 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	430 defer insertStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	431
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	432 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	433 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	434 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	435 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	436 defer trackStmt.Close()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	437
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	438 var ids []int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	439
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	440 lines:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	441 for line := 1; ; line++ {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	442
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	443 row, err := r.Read()
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	444 switch {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	445 case err == io.EOF \|\| len(row) == 0:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	446 break lines
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	447 case err != nil:
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	448 return nil, fmt.Errorf("CSV parsing failed: %v", err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	449 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	450
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	451 location, err := models.IsrsFromString(row[locationIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	452 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	453 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	454 "Invalid ISRS location code in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	455 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	456 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	457
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	458 validFromTime, err := parseDate(row[validFromIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	459 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	460 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	461 "Invalid 'valid_from' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	462 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	463 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	464 validToTime, err := parseDate(row[validToIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	465 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	466 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	467 "Invalid 'valid_to' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	468 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	469 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	470
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	471 validFrom := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	472 Time: validFromTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	473 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	474 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	475
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	476 validTo := pgtype.Timestamptz{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	477 Time: validToTime,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	478 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	479 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	480
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	481 validity := pgtype.Tstzrange{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	482 Lower: validFrom,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	483 Upper: validTo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	484 LowerType: pgtype.Inclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	485 UpperType: pgtype.Exclusive,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	486 Status: pgtype.Present,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	487 }
2078 dacf79a0658e Waterway profile import: Parse header line. TODO: Parse the body and store the data into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2072 diff changeset	488
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	489 lnwl, err := parseFloat64(row[lnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	490 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	491 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	492 "Invalid 'lnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	493 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	494 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	495 mwl, err := parseFloat64(row[mwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	496 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	497 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	498 "Invalid 'mwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	499 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	500 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	501 hnwl, err := parseFloat64(row[hnwlIdx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	502 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	503 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	504 "Invalid 'hnwl' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	505 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	506 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	507 fe30, err := parseFloat64(row[fe30Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	508 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	509 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	510 "Invalid 'fe30' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	511 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	512 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	513 fe100, err := parseFloat64(row[fe100Idx])
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	514 if err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	515 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	516 "Invalid 'fe100' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	517 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	518 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	519
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	520 var dateInfo time.Time
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	521
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	522 if di := row[dateInfoIdx]; di == "" {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	523 dateInfo = start
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	524 } else if dateInfo, err = parseDate(di); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	525 return nil, fmt.Errorf(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	526 "Invalid 'date_info' value in line %d: %v",
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	527 line, err)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	528 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	529
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	530 source := row[sourceIdx]
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	531
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	532 var id int64
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	533 if err := insertStmt.QueryRowContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	534 ctx,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	535 location.CountryCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	536 location.LoCode,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	537 location.FairwaySection,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	538 location.Orc,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	539 location.Hectometre,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	540 &validity,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	541 lnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	542 mwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	543 hnwl,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	544 fe30,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	545 fe100,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	546 dateInfo,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	547 source,
2086 6096ec4951f8 Waterway profiles imports: Download the geometries from WPS and join them with the CSV data. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2084 diff changeset	548 epsg,
2079 9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	549 ).Scan(&id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	550 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	551 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	552
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	553 if _, err := trackStmt.ExecContext(
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	554 ctx, importID, "waterway.waterway_profiles", id); err != nil {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	555 return nil, err
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	556 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	557 ids = append(ids, id)
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	558 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	559 if len(ids) == 0 {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	560 return nil, UnchangedError("No new entries in waterway profiles.")
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	561 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	562
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	563 feedback.Info("%d new entries in waterway profiles.", len(ids))
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	564
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	565 summary := struct {
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	566 IDs []int64 `json:"ids"`
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	567 }{
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	568 IDs: ids,
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	569 }
9318973487a1 Waterway profiles import: Implemented the parsing of the data lines of the CSV file and stored the values into the database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2078 diff changeset	570 return &summary, nil
2072 b4d8d320feab Waterway profile import: Added import stub and changed database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	571 }

Mercurial > gemma

annotate pkg/imports/wp.go @ 2086:6096ec4951f8