Mercurial > gemma

annotate pkg/imports/erdms.go @ 5591:0011f50cf216 surveysperbottleneckid

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.
author Sascha Wilde <wilde@sha-bang.de>
date Wed, 06 Apr 2022 13:30:29 +0200
parents 5f47eeea988d
children e1936db6db8e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
3 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
6 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
10 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
11 // Author(s):
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
14
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
15 package imports
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
16
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
17 import (
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
18 "context"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
19 "database/sql"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
20 "fmt"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
21 "strings"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
22
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3551
diff changeset
23 "gemma.intevation.de/gemma/pkg/log"
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
24 "gemma.intevation.de/gemma/pkg/soap"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/soap/erdms"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
26 )
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
27
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
28 const (
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
29 selectUserCountriesSQL = `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
30 SELECT DISTINCT country FROM users.list_users
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
31 WHERE country <> '--'
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
32 `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
33 )
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
34
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
35 func userCountries(ctx context.Context, conn *sql.Conn) ([]string, error) {
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
36 rows, err := conn.QueryContext(ctx, selectUserCountriesSQL)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
37 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
38 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
39 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
40 defer rows.Close()
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
41
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
42 var countries []string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
43
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
44 for rows.Next() {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
45 var country string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
46 if err = rows.Scan(&country); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
47 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
48 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
49 countries = append(countries, country)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
50 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
51
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
52 if err := rows.Err(); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
53 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
54 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
55
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
56 return countries, nil
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
57 }
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
58
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
59 func getRisData(
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
60 ctx context.Context,
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
61 conn *sql.Conn,
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
62 feedback Feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
63 username string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
64 password string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
65 URL string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
66 insecure bool,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
67 funcode string,
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
68 ) ([]*erdms.GetRisDataXMLResponse, []string, error) {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
69
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
70 countries, err := userCountries(ctx, conn)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
71 if err != nil {
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
72 return nil, nil, err
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
73 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
74
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
75 var auth *soap.BasicAuth
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
76 if username != "" {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
77 auth = &soap.BasicAuth{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
78 Login: username,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
79 Password: password,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
80 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
81 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
82
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
83 client := erdms.NewRefService(URL, insecure, auth)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
84
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
85 var responseData []*erdms.GetRisDataXMLResponse
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
86 for _, country := range countries {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
87
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
88 feedback.Info("Request RIS index for country %s", country)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
89
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
90 request := &erdms.GetRisDataXML{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
91 GetRisDataXMLType: &erdms.GetRisDataXMLType{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
92 Subcode: erdms.NoNS{Text: country + "%"},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
93 Funcode: erdms.NoNS{Text: funcode},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
94 },
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
95 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
96
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
97 const maxTries = 3
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
98
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
99 tries := 0
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
100
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
101 again:
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
102 data, err := client.GetRisDataXML(request)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
103 if err != nil {
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
104 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3551
diff changeset
105 log.Warnln("warn: ERDMS SOAP request timed out. Trying again.")
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
106 tries++
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
107 goto again
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
108 }
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
109 return nil, nil, fmt.Errorf(
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
110 "Error requesting ERDMS service: %v", err)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
111 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
112 responseData = append(responseData, data)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
113 }
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
114
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
115 feedback.Info("Import data for countries: %s.",
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
116 strings.Join(countries, ", "))
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
117
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
118 return responseData, countries, nil
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
119 }