Mercurial > gemma

annotate pkg/imports/fa.go @ 4515:7465b6244d5e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
FA import: Code clean-up for handling default LOS.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Fri, 04 Oct 2019 17:37:51 +0200
parents 9b1e5a3ed3cc
children 1e96ff2da1f3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
3 //
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
6 //
4072
1eb39e9e8ec2 Adapted fwa-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
7 // Copyright (C) 2018,2019 by via donau
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
10 //
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
11 // Author(s):
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
12 // * Raimund Renkert <raimund.renkert@intevation.de>
2203
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2202
diff changeset
13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
4072
1eb39e9e8ec2 Adapted fwa-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
14 // * Sascha Wilde <wilde@intevation.de>
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1676
diff changeset
15
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
16 package imports
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
17
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
18 import (
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
19 "context"
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
20 "database/sql"
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
21 "errors"
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
22 "fmt"
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
23 "sort"
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
24 "time"
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
25
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
26 "github.com/jackc/pgx/pgtype"
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
27
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
28 "gemma.intevation.de/gemma/pkg/soap/ifaf"
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
29 )
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
30
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
31 // FairwayAvailability imports fairway availability data
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
32 // from an IFAF SOAP service.
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
33 type FairwayAvailability struct {
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
34 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
35 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
36 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
37 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
38 Insecure bool `json:"insecure"`
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
39 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
40
4281
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
41 const (
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
42 // FAJobKind is import queue type identifier.
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
43 FAJobKind JobKind = "fa"
4283
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
44 )
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
45
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
46 const (
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
47 // maxHistoryDays is the numbers of days to look back.
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
48 maxHistoryDays = 7
4281
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
49 )
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
50
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
51 const (
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
52 listBottlenecksSQL = `
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
53 SELECT DISTINCT
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
54 bottleneck_id
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
55 FROM waterway.bottlenecks
4158
5466562cca60 Remove utility function with possibly bad performance impact
Tom Gottfried <tom@intevation.de>
parents: 4081
diff changeset
56 WHERE responsible_country = (
5466562cca60 Remove utility function with possibly bad performance impact
Tom Gottfried <tom@intevation.de>
parents: 4081
diff changeset
57 SELECT country FROM users.list_users WHERE username = current_user)
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
58 AND staging_done = true
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
59 ORDER BY bottleneck_id
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
60 `
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
61
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
62 latestMeasureDateSQL = `
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
63 SELECT
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
64 measure_date
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
65 FROM waterway.effective_fairway_availability
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
66 ORDER BY measure_date DESC LIMIT 1
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
67 `
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
68
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
69 insertFASQL = `
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
70 INSERT INTO waterway.fairway_availability (
4461
7128741faeb9 Changed fa data to accept "position" as strings, not a 2char "position_code"
Sascha Wilde <wilde@intevation.de>
parents: 4283
diff changeset
71 position,
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
72 bottleneck_id,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
73 surdat,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
74 critical,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
75 date_info,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
76 source_organization
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
77 ) VALUES (
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
78 $1,
4072
1eb39e9e8ec2 Adapted fwa-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
79 $2,
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
80 $3,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
81 $4,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
82 $5,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
83 $6
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
84 ) ON CONFLICT (bottleneck_id, surdat) DO UPDATE SET
4461
7128741faeb9 Changed fa data to accept "position" as strings, not a 2char "position_code"
Sascha Wilde <wilde@intevation.de>
parents: 4283
diff changeset
85 position = EXCLUDED.position,
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
86 critical = EXCLUDED.critical,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
87 date_info = EXCLUDED.date_info,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
88 source_organization = EXCLUDED.source_organization
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
89 RETURNING id`
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
90
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
91 insertBnPdfsSQL = `
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
92 INSERT INTO waterway.bottleneck_pdfs (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
93 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
94 profile_pdf_filename,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
95 profile_pdf_url,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
96 pdf_generation_date,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
97 source_organization
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
98 ) VALUES (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
99 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
100 $2,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
101 $3,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
102 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
103 $5
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
104 ) ON CONFLICT ON CONSTRAINT bottleneck_pdfs_pkey DO NOTHING`
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
105
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
106 insertEFASQL = `
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
107 INSERT INTO waterway.effective_fairway_availability (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
108 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
109 measure_date,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
110 level_of_service,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
111 available_depth_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
112 available_width_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
113 water_level_value,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
114 measure_type,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
115 source_organization,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
116 forecast_generation_time,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
117 value_lifetime
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
118 ) VALUES (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
119 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
120 $2,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
121 (SELECT
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
122 level_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
123 FROM levels_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
124 WHERE name = $3),
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
125 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
126 $5,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
127 $6,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
128 $7,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
129 $8,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
130 $9,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
131 $10
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
132 ) ON CONFLICT ON CONSTRAINT effective_fairway_availability_pkey DO NOTHING`
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
133
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
134 insertFAVSQL = `
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
135 INSERT INTO waterway.fa_reference_values (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
136 fairway_availability_id,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
137 level_of_service,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
138 fairway_depth,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
139 fairway_width,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
140 fairway_radius,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
141 shallowest_spot
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
142 ) VALUES (
1663
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
143 $1,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
144 (SELECT
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
145 level_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
146 FROM levels_of_service
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
147 WHERE name = $2),
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
148 $3,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
149 $4,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
150 $5,
10e3dd3b9363 Fairway availability import: Don't use tabs in SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1662
diff changeset
151 ST_MakePoint($6, $7)::geography
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
152 )ON CONFLICT ON CONSTRAINT fa_reference_values_pkey DO NOTHING`
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
153 )
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
154
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
155 type faJobCreator struct{}
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
156
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
157 func init() {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
158 RegisterJobCreator(FAJobKind, faJobCreator{})
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
159 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
160
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
161 func (faJobCreator) Description() string {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
162 return "fairway availability"
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
163 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
164
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
165 func (faJobCreator) Create() Job { return new(FairwayAvailability) }
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
166
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2253
diff changeset
167 func (faJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2253
diff changeset
168 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2253
diff changeset
169 {"effective_fairway_availability", "fa_reference_values",
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2253
diff changeset
170 "bottleneck_pdfs", "fairway_availability"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2253
diff changeset
171 {"bottlenecks", "levels_of_service"},
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
172 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
173 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
174
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
175 func (faJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
176
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
177 // StageDone moves the imported fairway availablities out of the staging area.
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
178 // Currently doing nothing.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1663
diff changeset
179 func (faJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
180 return nil
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
181 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
182
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
183 // CleanUp of a fairway availablities import is a NOP.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1663
diff changeset
184 func (*FairwayAvailability) CleanUp() error { return nil }
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
185
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
186 type bottlenecks []string
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
187
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
188 func (bns bottlenecks) contains(bn string) bool {
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
189 idx := sort.SearchStrings(bns, bn)
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
190 return idx < len(bns) && bns[idx] == bn
2016
25967829cf00 Started to simplify the import models.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1756
diff changeset
191 }
25967829cf00 Started to simplify the import models.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1756
diff changeset
192
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
193 func loadBottleneckCountries(ctx context.Context, tx *sql.Tx) (bottlenecks, error) {
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
194
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
195 // Get available bottlenecks from database for use as filter in SOAP request
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
196 rows, err := tx.QueryContext(ctx, listBottlenecksSQL)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
197 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
198 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
199 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
200 defer rows.Close()
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
201
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
202 var bns bottlenecks
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
203
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
204 for rows.Next() {
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
205 var bn string
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
206 if err = rows.Scan(&bn); err != nil {
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
207 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
208 }
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
209 bns = append(bns, bn)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
210 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
211 if err = rows.Err(); err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
212 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
213 }
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
214
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
215 return bns, nil
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
216 }
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
217
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
218 func latestDate(ctx context.Context, tx *sql.Tx) (pgtype.Timestamp, error) {
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
219 var date pgtype.Timestamp
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
220 err := tx.QueryRowContext(ctx, latestMeasureDateSQL).Scan(&date)
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
221 switch {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
222 case err == sql.ErrNoRows:
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
223 date = pgtype.Timestamp{
4283
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
224 Time: time.Now().AddDate(0, 0, -maxHistoryDays),
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
225 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
226 case err != nil:
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
227 return pgtype.Timestamp{}, err
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
228 }
4281
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
229 // Limit request range to MaxHistoryDays. Otherwise the Request might
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
230 // fail, e.g. the AT service has an upper liimit of 10000 results.
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
231 //
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
232 // FIXME: the better solution would be to detect such errors and
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
233 // dynamically and implement some kind of chunking...
4283
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
234 if time.Since(date.Time).Hours() > maxHistoryDays*24 {
4281
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
235 date = pgtype.Timestamp{
4283
c811d5da69bd Use untyped constant and let the compiler do the work.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4281
diff changeset
236 Time: time.Now().AddDate(0, 0, -maxHistoryDays),
4281
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
237 }
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
238 }
c470d2202823 Limit time range requested from fa services.
Sascha Wilde <wilde@intevation.de>
parents: 4158
diff changeset
239
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
240 return date, nil
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
241 }
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
242
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
243 func storeFairwayAvailability(
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
244 ctx context.Context,
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
245 conn *sql.Conn,
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
246 feedback Feedback,
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
247 fetch func(context.Context, *sql.Tx, bottlenecks) ([]*ifaf.FairwayAvailability, error),
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
248 ) (interface{}, error) {
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
249
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
250 start := time.Now()
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
251
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
252 tx, err := conn.BeginTx(ctx, nil)
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
253 if err != nil {
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
254 return nil, err
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
255 }
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
256 defer tx.Rollback()
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
257
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
258 bns, err := loadBottleneckCountries(ctx, tx)
2198
4db1fa4f049c Fairway availabilty import: Fixed row query leak.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2196
diff changeset
259 if err != nil {
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
260 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
261 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
262
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
263 fas, err := fetch(ctx, tx, bns)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
264 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
265 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
266 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
267
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
268 faids, err := doForFAs(ctx, bns, fas, tx, feedback)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
269 if err != nil {
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
270 return nil, fmt.Errorf("Error processing data: %v", err)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
271 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
272 if len(faids) == 0 {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
273 feedback.Info("No new fairway availablity data found")
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
274 return nil, UnchangedError("No new fairway availablity data found")
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
275 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
276 feedback.Info("Processed %d fairway availabilities", len(faids))
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
277
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2203
diff changeset
278 if err = tx.Commit(); err != nil {
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
279 feedback.Info(
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
280 "Importing fairway availabilities failed after %s", time.Since(start))
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
281 return nil, err
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
282 }
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2203
diff changeset
283 feedback.Info(
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2203
diff changeset
284 "Importing fairway availabilities successfully took %s", time.Since(start))
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
285
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
286 // TODO: needs to be filled more useful.
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
287 summary := struct {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
288 FairwayAvailabilities []string `json:"fairwayAvailabilities"`
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
289 }{
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
290 FairwayAvailabilities: faids,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
291 }
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
292 return &summary, nil
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
293 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
294
4515
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
295 // defaultLOS defaults to LOS3 when no expicit LOS is given.
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
296 func defaultLOS(los *ifaf.LosEnum) ifaf.LosEnum {
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
297 if los == nil {
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
298 return ifaf.LosEnumLOS3
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
299 }
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
300 return *los
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
301 }
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
302
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
303 func doForFAs(
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
304 ctx context.Context,
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
305 bnIds bottlenecks,
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
306 fas []*ifaf.FairwayAvailability,
2200
64147a137e0a Fairway availability import: Run all db ops in same transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2198
diff changeset
307 tx *sql.Tx,
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
308 feedback Feedback,
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
309 ) ([]string, error) {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
310
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
311 insertFAStmt, err := tx.PrepareContext(ctx, insertFASQL)
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
312 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
313 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
314 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
315 defer insertFAStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
316 insertBnPdfsStmt, err := tx.PrepareContext(ctx, insertBnPdfsSQL)
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
317 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
318 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
319 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
320 defer insertBnPdfsStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
321 insertEFAStmt, err := tx.PrepareContext(ctx, insertEFASQL)
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
322 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
323 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
324 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
325 defer insertEFAStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
326 insertFAVStmt, err := tx.PrepareContext(ctx, insertFAVSQL)
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
327 if err != nil {
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
328 return nil, err
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
329 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
330 defer insertFAVStmt.Close()
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
331
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1676
diff changeset
332 var faIDs []string
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1676
diff changeset
333 var faID int64
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
334 feedback.Info("Found %d fairway availabilities", len(fas))
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
335 for _, faRes := range fas {
4072
1eb39e9e8ec2 Adapted fwa-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
336 // FIXME: The following test is propably unneccessary as already
1eb39e9e8ec2 Adapted fwa-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
337 // done by DB constraints... [sw]
2201
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
338 if !bnIds.contains(faRes.Bottleneck_id) {
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
339 feedback.Warn("Bottleneck %s not found in database.", faRes.Bottleneck_id)
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
340 continue
cae0b597aefc Fairway availabilty import: There is no need to fetch the responsibility country from the database. Only the bottleneck_ids are needed.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2200
diff changeset
341 }
4079
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
342 err = insertFAStmt.QueryRowContext(
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
343 ctx,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
344 faRes.POSITION,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
345 faRes.Bottleneck_id,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
346 faRes.SURDAT,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
347 faRes.Critical,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
348 faRes.Date_Info,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
349 faRes.Source,
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
350 ).Scan(&faID)
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
351 if err != nil {
7fb75deff16b Allow fairway availability data to be updated
Tom Gottfried <tom@intevation.de>
parents: 3666
diff changeset
352 return nil, err
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
353 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
354 feedback.Info("Processing for Bottleneck %s", faRes.Bottleneck_id)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
355 faIDs = append(faIDs, faRes.Bottleneck_id)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
356 if faRes.Bottleneck_PDFs != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
357 bnPdfCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
358 for _, bnPdfs := range faRes.Bottleneck_PDFs.PdfInfo {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
359 res, err := insertBnPdfsStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
360 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
361 faID,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
362 bnPdfs.ProfilePdfFilename,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
363 bnPdfs.ProfilePdfURL,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
364 bnPdfs.PDF_Generation_Date,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
365 bnPdfs.Source,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
366 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
367 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
368 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
369 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
370 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
371 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
372 bnPdfCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
373 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
374 bnPdfCount++
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
375 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
376 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
377 feedback.Info("Add %d Pdfs", bnPdfCount)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
378 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
379 if faRes.Effective_fairway_availability != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
380 efaCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
381 for _, efa := range faRes.Effective_fairway_availability.EffectiveFairwayAvailability {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
382 fgt := efa.Forecast_generation_time
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
383 if efa.Forecast_generation_time.Status == pgtype.Undefined {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
384 fgt = pgtype.Timestamp{
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
385 Status: pgtype.Null,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
386 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
387 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
388 res, err := insertEFAStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
389 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
390 faID,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
391 efa.Measure_date,
4515
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
392 string(defaultLOS(efa.Level_of_Service)),
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
393 efa.Available_depth_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
394 efa.Available_width_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
395 efa.Water_level_value,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
396 efa.Measure_type,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
397 efa.Source,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
398 fgt.Get(),
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
399 efa.Value_lifetime,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
400 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
401 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
402 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
403 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
404 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
405 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
406 efaCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
407 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
408 efaCount++
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
409 }
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
410 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
411 feedback.Info("Add %d Effective Fairway Availability", efaCount)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
412 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
413
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
414 if faRes.Reference_values != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
415 rvCount := 0
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
416 for _, fav := range faRes.Reference_values.ReferenceValue {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
417 res, err := insertFAVStmt.ExecContext(
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
418 ctx,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
419 faID,
4515
7465b6244d5e FA import: Code clean-up for handling default LOS.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4513
diff changeset
420 string(defaultLOS(fav.Level_of_Service)),
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
421 fav.Fairway_depth,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
422 fav.Fairway_width,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
423 fav.Fairway_radius,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
424 fav.Shallowest_spot_Lat,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
425 fav.Shallowest_spot_Lon,
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
426 )
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
427 if err != nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
428 return nil, err
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
429 }
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
430 affected, err := res.RowsAffected()
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
431 if err == nil {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
432 rvCount += int(affected)
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
433 } else {
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
434 rvCount++
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
435 }
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
436 }
1755
d3fe20a13339 Import: Filter fairway availabilities by period and resolve insert confilcts.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1754
diff changeset
437 feedback.Info("Add %d Reference Values", rvCount)
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
438 }
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
439 }
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1676
diff changeset
440 return faIDs, nil
1662
d8ca44615bfc Implemented first version of fairway availability import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
441 }
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
442
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
443 // Do executes the actual fairway availability import.
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
444 func (fa *FairwayAvailability) Do(
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
445 ctx context.Context,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
446 importID int64,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
447 conn *sql.Conn,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
448 feedback Feedback,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
449 ) (interface{}, error) {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
450
2203
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2202
diff changeset
451 fetch := func(
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2202
diff changeset
452 ctx context.Context,
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2202
diff changeset
453 tx *sql.Tx, bns bottlenecks,
8d1a945d0c3b Uploaded fairway availabilty import: Implemented in terms of normal fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2202
diff changeset
454 ) ([]*ifaf.FairwayAvailability, error) {
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
455
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
456 latest, err := latestDate(ctx, tx)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
457 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
458 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
459 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
460
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
461 client := ifaf.NewFairwayAvailabilityService(fa.URL, fa.Insecure, nil)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
462
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
463 period := ifaf.RequestedPeriod{
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
464 Date_start: latest.Time,
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
465 Date_end: time.Now(),
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
466 }
2202
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
467
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
468 ids := ifaf.ArrayOfString{String: bns}
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
469
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
470 req := &ifaf.Get_bottleneck_fa{
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
471 Bottleneck_id: &ids,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
472 Period: &period,
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
473 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
474 resp, err := client.Get_bottleneck_fa(req)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
475 if err != nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
476 return nil, err
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
477 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
478
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
479 if resp.Get_bottleneck_faResult == nil {
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
480 return nil, errors.New("no fairway availabilities found")
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
481 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
482
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
483 result := resp.Get_bottleneck_faResult
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
484 return result.FairwayAvailability, nil
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
485 }
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
486
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
487 return storeFairwayAvailability(ctx, conn, feedback, fetch)
0aee7d4954ae Fairway availabilty import: Re-factored to be the base for the uploaded fairway availabilty import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2201
diff changeset
488 }