Mercurial > gemma

annotate pkg/imports/erdms.go @ 5490:5f47eeea988d logging

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Use own logging package.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Mon, 20 Sep 2021 17:45:39 +0200
parents a5448426e4e2
children e1936db6db8e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
3 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
6 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
10 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
11 // Author(s):
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
14
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
15 package imports
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
16
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
17 import (
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
18 "context"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
19 "database/sql"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
20 "fmt"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
21 "strings"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
22
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3551
diff changeset
23 "gemma.intevation.de/gemma/pkg/log"
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
24 "gemma.intevation.de/gemma/pkg/soap"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/soap/erdms"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
26 )
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
27
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
28 const (
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
29 selectUserCountriesSQL = `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
30 SELECT DISTINCT country FROM users.list_users
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
31 WHERE country <> '--'
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
32 `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
33 )
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
34
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
35 func userCountries(ctx context.Context, conn *sql.Conn) ([]string, error) {
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
36 rows, err := conn.QueryContext(ctx, selectUserCountriesSQL)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
37 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
38 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
39 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
40 defer rows.Close()
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
41
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
42 var countries []string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
43
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
44 for rows.Next() {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
45 var country string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
46 if err = rows.Scan(&country); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
47 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
48 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
49 countries = append(countries, country)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
50 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
51
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
52 if err := rows.Err(); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
53 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
54 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
55
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
56 return countries, nil
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
57 }
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
58
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
59 func getRisData(
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
60 ctx context.Context,
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
61 conn *sql.Conn,
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
62 feedback Feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
63 username string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
64 password string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
65 URL string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
66 insecure bool,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
67 funcode string,
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
68 ) ([]*erdms.GetRisDataXMLResponse, []string, error) {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
69
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
70 countries, err := userCountries(ctx, conn)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
71 if err != nil {
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
72 return nil, nil, err
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
73 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
74
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
75 var auth *soap.BasicAuth
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
76 if username != "" {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
77 auth = &soap.BasicAuth{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
78 Login: username,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
79 Password: password,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
80 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
81 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
82
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
83 client := erdms.NewRefService(URL, insecure, auth)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
84
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
85 var responseData []*erdms.GetRisDataXMLResponse
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
86 for _, country := range countries {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
87
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
88 feedback.Info("Request RIS index for country %s", country)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
89
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
90 request := &erdms.GetRisDataXML{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
91 GetRisDataXMLType: &erdms.GetRisDataXMLType{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
92 Subcode: erdms.NoNS{Text: country + "%"},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
93 Funcode: erdms.NoNS{Text: funcode},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
94 },
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
95 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
96
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
97 const maxTries = 3
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
98
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
99 tries := 0
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
100
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
101 again:
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
102 data, err := client.GetRisDataXML(request)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
103 if err != nil {
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
104 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3551
diff changeset
105 log.Warnln("warn: ERDMS SOAP request timed out. Trying again.")
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
106 tries++
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
107 goto again
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
108 }
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
109 return nil, nil, fmt.Errorf(
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
110 "Error requesting ERDMS service: %v", err)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
111 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
112 responseData = append(responseData, data)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
113 }
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
114
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
115 feedback.Info("Import data for countries: %s.",
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
116 strings.Join(countries, ", "))
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
117
3551
a5448426e4e2 Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents: 3163
diff changeset
118 return responseData, countries, nil
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
119 }