gemma: pkg/imports/fa.go annotate

annotate pkg/imports/fa.go @ 4281:c470d2202823

Limit time range requested from fa services. This is a workaround, so that services which limit the amount of results in there responses don't bail out, when the latest data is very old.

author	Sascha Wilde <wilde@intevation.de>
date	Thu, 29 Aug 2019 16:59:47 +0200
parents	5466562cca60
children	c811d5da69bd

rev	line source
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
4072 1eb39e9e8ec2 Adapted fwa-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	7 // Copyright (C) 2018,2019 by via donau
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2203 8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2202 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
4072 1eb39e9e8ec2 Adapted fwa-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	14 // * Sascha Wilde <wilde@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1676 diff changeset	15
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	21 "errors"
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	22 "fmt"
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	23 "sort"
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	24 "time"
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	26 "github.com/jackc/pgx/pgtype"
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	27
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	28 "gemma.intevation.de/gemma/pkg/soap/ifaf"
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29 )
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	30
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	31 // FairwayAvailability imports fairway availability data
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	32 // from an IFAF SOAP service.
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33 type FairwayAvailability struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	34 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	36 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	37 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 Insecure bool `json:"insecure"`
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	39 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	40
4281 c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	41 const (
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	42 // FAJobKind is import queue type identifier.
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	43 FAJobKind JobKind = "fa"
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	44 // Upper limit for days to request
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	45 MaxHistoryDays int = 7
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	46 )
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	47
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	48 const (
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	49 listBottlenecksSQL = `
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	50 SELECT DISTINCT
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	51 bottleneck_id
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52 FROM waterway.bottlenecks
4158 5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4081 diff changeset	53 WHERE responsible_country = (
5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4081 diff changeset	54 SELECT country FROM users.list_users WHERE username = current_user)
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	55 AND staging_done = true
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	56 ORDER BY bottleneck_id
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	57 `
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	58
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	59 latestMeasureDateSQL = `
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	60 SELECT
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	61 measure_date
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	62 FROM waterway.effective_fairway_availability
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	63 ORDER BY measure_date DESC LIMIT 1
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	64 `
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	65
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	66 insertFASQL = `
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	67 INSERT INTO waterway.fairway_availability (
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	68 position_code,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 bottleneck_id,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 surdat,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 critical,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 date_info,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 source_organization
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 ) VALUES (
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	75 $1,
4072 1eb39e9e8ec2 Adapted fwa-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	76 $2,
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 $3,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	78 $4,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 $5,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	80 $6
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	81 ) ON CONFLICT (bottleneck_id, surdat) DO UPDATE SET
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	82 position_code = EXCLUDED.position_code,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	83 critical = EXCLUDED.critical,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	84 date_info = EXCLUDED.date_info,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	85 source_organization = EXCLUDED.source_organization
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 RETURNING id`
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 insertBnPdfsSQL = `
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 INSERT INTO waterway.bottleneck_pdfs (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	90 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	91 profile_pdf_filename,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	92 profile_pdf_url,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	93 pdf_generation_date,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	94 source_organization
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	95 ) VALUES (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	96 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	97 $2,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	98 $3,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	99 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	100 $5
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	101 ) ON CONFLICT ON CONSTRAINT bottleneck_pdfs_pkey DO NOTHING`
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	102
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	103 insertEFASQL = `
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	104 INSERT INTO waterway.effective_fairway_availability (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	105 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	106 measure_date,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	107 level_of_service,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	108 available_depth_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	109 available_width_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	110 water_level_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	111 measure_type,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	112 source_organization,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	113 forecast_generation_time,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	114 value_lifetime
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	115 ) VALUES (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	116 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	117 $2,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	118 (SELECT
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	119 level_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	120 FROM levels_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	121 WHERE name = $3),
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	122 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	123 $5,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	124 $6,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	125 $7,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	126 $8,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	127 $9,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	128 $10
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	129 ) ON CONFLICT ON CONSTRAINT effective_fairway_availability_pkey DO NOTHING`
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	130
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131 insertFAVSQL = `
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132 INSERT INTO waterway.fa_reference_values (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	133 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	134 level_of_service,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	135 fairway_depth,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	136 fairway_width,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	137 fairway_radius,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	138 shallowest_spot
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139 ) VALUES (
1663 10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	140 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	141 (SELECT
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	142 level_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	143 FROM levels_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	144 WHERE name = $2),
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	145 $3,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	146 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	147 $5,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1662 diff changeset	148 ST_MakePoint($6, $7)::geography
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	149 )ON CONFLICT ON CONSTRAINT fa_reference_values_pkey DO NOTHING`
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	150 )
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	151
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 type faJobCreator struct{}
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154 func init() {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	155 RegisterJobCreator(FAJobKind, faJobCreator{})
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	156 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	157
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	158 func (faJobCreator) Description() string {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	159 return "fairway availability"
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	160 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	161
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	162 func (faJobCreator) Create() Job { return new(FairwayAvailability) }
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	163
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2253 diff changeset	164 func (faJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2253 diff changeset	165 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2253 diff changeset	166 {"effective_fairway_availability", "fa_reference_values",
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2253 diff changeset	167 "bottleneck_pdfs", "fairway_availability"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2253 diff changeset	168 {"bottlenecks", "levels_of_service"},
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	169 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	170 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	171
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	172 func (faJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	173
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	174 // StageDone moves the imported fairway availablities out of the staging area.
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	175 // Currently doing nothing.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1663 diff changeset	176 func (faJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	177 return nil
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	178 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	179
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	180 // CleanUp of a fairway availablities import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1663 diff changeset	181 func (*FairwayAvailability) CleanUp() error { return nil }
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	182
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	183 type bottlenecks []string
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	184
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	185 func (bns bottlenecks) contains(bn string) bool {
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	186 idx := sort.SearchStrings(bns, bn)
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	187 return idx < len(bns) && bns[idx] == bn
2016 25967829cf00 Started to simplify the import models. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1756 diff changeset	188 }
25967829cf00 Started to simplify the import models. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1756 diff changeset	189
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	190 func loadBottleneckCountries(ctx context.Context, tx *sql.Tx) (bottlenecks, error) {
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	191
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	192 // Get available bottlenecks from database for use as filter in SOAP request
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	193 rows, err := tx.QueryContext(ctx, listBottlenecksSQL)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	194 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	195 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	196 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	197 defer rows.Close()
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	198
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	199 var bns bottlenecks
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	200
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	201 for rows.Next() {
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	202 var bn string
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	203 if err = rows.Scan(&bn); err != nil {
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	204 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	205 }
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	206 bns = append(bns, bn)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	207 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	208 if err = rows.Err(); err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	209 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	210 }
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	211
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	212 return bns, nil
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	213 }
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	214
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	215 func latestDate(ctx context.Context, tx *sql.Tx) (pgtype.Timestamp, error) {
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	216 var date pgtype.Timestamp
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	217 err := tx.QueryRowContext(ctx, latestMeasureDateSQL).Scan(&date)
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	218 switch {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	219 case err == sql.ErrNoRows:
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	220 date = pgtype.Timestamp{
4281 c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	221 Time: time.Now().AddDate(0, 0, MaxHistoryDays*-1),
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	222 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	223 case err != nil:
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	224 return pgtype.Timestamp{}, err
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	225 }
4281 c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	226 // Limit request range to MaxHistoryDays. Otherwise the Request might
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	227 // fail, e.g. the AT service has an upper liimit of 10000 results.
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	228 //
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	229 // FIXME: the better solution would be to detect such errors and
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	230 // dynamically and implement some kind of chunking...
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	231 if time.Since(date.Time).Hours() > float64(MaxHistoryDays*24) {
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	232 date = pgtype.Timestamp{
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	233 Time: time.Now().AddDate(0, 0, MaxHistoryDays*-1),
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	234 }
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	235 }
c470d2202823 Limit time range requested from fa services. Sascha Wilde <wilde@intevation.de> parents: 4158 diff changeset	236
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	237 return date, nil
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	238 }
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	239
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	240 func storeFairwayAvailability(
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	241 ctx context.Context,
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	242 conn *sql.Conn,
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	243 feedback Feedback,
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	244 fetch func(context.Context, sql.Tx, bottlenecks) ([]ifaf.FairwayAvailability, error),
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	245 ) (interface{}, error) {
4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	246
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	247 start := time.Now()
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	248
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	249 tx, err := conn.BeginTx(ctx, nil)
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	250 if err != nil {
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	251 return nil, err
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	252 }
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	253 defer tx.Rollback()
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	254
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	255 bns, err := loadBottleneckCountries(ctx, tx)
2198 4db1fa4f049c Fairway availabilty import: Fixed row query leak. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2196 diff changeset	256 if err != nil {
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	257 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	258 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	259
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	260 fas, err := fetch(ctx, tx, bns)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	261 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	262 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	263 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	264
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	265 faids, err := doForFAs(ctx, bns, fas, tx, feedback)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	266 if err != nil {
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	267 return nil, fmt.Errorf("Error processing data: %v", err)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	268 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	269 if len(faids) == 0 {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	270 feedback.Info("No new fairway availablity data found")
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	271 return nil, UnchangedError("No new fairway availablity data found")
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	272 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	273 feedback.Info("Processed %d fairway availabilities", len(faids))
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	274
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2203 diff changeset	275 if err = tx.Commit(); err != nil {
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	276 feedback.Info(
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	277 "Importing fairway availabilities failed after %s", time.Since(start))
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	278 return nil, err
64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	279 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2203 diff changeset	280 feedback.Info(
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2203 diff changeset	281 "Importing fairway availabilities successfully took %s", time.Since(start))
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	282
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	283 // TODO: needs to be filled more useful.
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	284 summary := struct {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	285 FairwayAvailabilities []string `json:"fairwayAvailabilities"`
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 }{
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 FairwayAvailabilities: faids,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 }
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	289 return &summary, nil
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	292 func doForFAs(
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 ctx context.Context,
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	294 bnIds bottlenecks,
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	295 fas []*ifaf.FairwayAvailability,
2200 64147a137e0a Fairway availability import: Run all db ops in same transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2198 diff changeset	296 tx *sql.Tx,
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	297 feedback Feedback,
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	298 ) ([]string, error) {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	299
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	300 insertFAStmt, err := tx.PrepareContext(ctx, insertFASQL)
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	301 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	302 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	303 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	304 defer insertFAStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	305 insertBnPdfsStmt, err := tx.PrepareContext(ctx, insertBnPdfsSQL)
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	306 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	307 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	308 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	309 defer insertBnPdfsStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	310 insertEFAStmt, err := tx.PrepareContext(ctx, insertEFASQL)
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	311 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	312 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	313 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	314 defer insertEFAStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	315 insertFAVStmt, err := tx.PrepareContext(ctx, insertFAVSQL)
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	316 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	317 return nil, err
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	318 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	319 defer insertFAVStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	320
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1676 diff changeset	321 var faIDs []string
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1676 diff changeset	322 var faID int64
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	323 feedback.Info("Found %d fairway availabilities", len(fas))
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	324 for _, faRes := range fas {
4072 1eb39e9e8ec2 Adapted fwa-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	325 // FIXME: The following test is propably unneccessary as already
1eb39e9e8ec2 Adapted fwa-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	326 // done by DB constraints... [sw]
2201 cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	327 if !bnIds.contains(faRes.Bottleneck_id) {
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	328 feedback.Warn("Bottleneck %s not found in database.", faRes.Bottleneck_id)
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	329 continue
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2200 diff changeset	330 }
4079 7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	331 err = insertFAStmt.QueryRowContext(
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	332 ctx,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	333 faRes.POSITION,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	334 faRes.Bottleneck_id,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	335 faRes.SURDAT,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	336 faRes.Critical,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	337 faRes.Date_Info,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	338 faRes.Source,
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	339 ).Scan(&faID)
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	340 if err != nil {
7fb75deff16b Allow fairway availability data to be updated Tom Gottfried <tom@intevation.de> parents: 3666 diff changeset	341 return nil, err
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	342 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	343 feedback.Info("Processing for Bottleneck %s", faRes.Bottleneck_id)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	344 faIDs = append(faIDs, faRes.Bottleneck_id)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	345 if faRes.Bottleneck_PDFs != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	346 bnPdfCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	347 for _, bnPdfs := range faRes.Bottleneck_PDFs.PdfInfo {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	348 res, err := insertBnPdfsStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	349 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	350 faID,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	351 bnPdfs.ProfilePdfFilename,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	352 bnPdfs.ProfilePdfURL,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	353 bnPdfs.PDF_Generation_Date,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	354 bnPdfs.Source,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	355 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	356 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	357 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	358 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	359 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	360 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	361 bnPdfCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	362 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	363 bnPdfCount++
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	364 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	365 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	366 feedback.Info("Add %d Pdfs", bnPdfCount)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	367 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	368 if faRes.Effective_fairway_availability != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	369 efaCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	370 for _, efa := range faRes.Effective_fairway_availability.EffectiveFairwayAvailability {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	371 los := efa.Level_of_Service
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	372 fgt := efa.Forecast_generation_time
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	373 if efa.Forecast_generation_time.Status == pgtype.Undefined {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	374 fgt = pgtype.Timestamp{
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	375 Status: pgtype.Null,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	376 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	377 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	378 res, err := insertEFAStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	379 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	380 faID,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	381 efa.Measure_date,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	382 string(*los),
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	383 efa.Available_depth_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	384 efa.Available_width_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	385 efa.Water_level_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	386 efa.Measure_type,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	387 efa.Source,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	388 fgt.Get(),
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	389 efa.Value_lifetime,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	390 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	391 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	392 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	393 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	394 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	395 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	396 efaCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	397 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	398 efaCount++
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	399 }
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	400 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	401 feedback.Info("Add %d Effective Fairway Availability", efaCount)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	402 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	403
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	404 if faRes.Reference_values != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	405 rvCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	406 for _, fav := range faRes.Reference_values.ReferenceValue {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	407 res, err := insertFAVStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	408 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	409 faID,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	410 fav.Level_of_Service,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	411 fav.Fairway_depth,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	412 fav.Fairway_width,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	413 fav.Fairway_radius,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	414 fav.Shallowest_spot_Lat,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	415 fav.Shallowest_spot_Lon,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	416 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	417 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	418 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	419 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	420 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	421 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	422 rvCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	423 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	424 rvCount++
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	425 }
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	426 }
1755 d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts. Raimund Renkert <raimund.renkert@intevation.de> parents: 1754 diff changeset	427 feedback.Info("Add %d Reference Values", rvCount)
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	428 }
d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	429 }
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1676 diff changeset	430 return faIDs, nil
1662 d8ca44615bfc Implemented first version of fairway availability import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	431 }
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	432
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	433 // Do executes the actual fairway availability import.
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	434 func (fa *FairwayAvailability) Do(
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	435 ctx context.Context,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	436 importID int64,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	437 conn *sql.Conn,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	438 feedback Feedback,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	439 ) (interface{}, error) {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	440
2203 8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2202 diff changeset	441 fetch := func(
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2202 diff changeset	442 ctx context.Context,
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2202 diff changeset	443 tx *sql.Tx, bns bottlenecks,
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2202 diff changeset	444 ) ([]*ifaf.FairwayAvailability, error) {
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	445
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	446 latest, err := latestDate(ctx, tx)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	447 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	448 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	449 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	450
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	451 client := ifaf.NewFairwayAvailabilityService(fa.URL, fa.Insecure, nil)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	452
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	453 period := ifaf.RequestedPeriod{
cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	454 Date_start: latest.Time,
cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	455 Date_end: time.Now(),
cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	456 }
2202 0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	457
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	458 ids := ifaf.ArrayOfString{String: bns}
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	459
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	460 req := &ifaf.Get_bottleneck_fa{
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	461 Bottleneck_id: &ids,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	462 Period: &period,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	463 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	464 resp, err := client.Get_bottleneck_fa(req)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	465 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	466 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	467 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	468
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	469 if resp.Get_bottleneck_faResult == nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	470 return nil, errors.New("no fairway availabilities found")
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	471 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	472
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	473 result := resp.Get_bottleneck_faResult
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	474 return result.FairwayAvailability, nil
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	475 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	476
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	477 return storeFairwayAvailability(ctx, conn, feedback, fetch)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2201 diff changeset	478 }

Mercurial > gemma

annotate pkg/imports/fa.go @ 4281:c470d2202823