Mercurial > gemma

annotate pkg/imports/erdms.go @ 3163:d9903cb34842

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Handle failing INSERTs gracefully during gauges import Using the special table EXCLUDED in INSERT statements makes functionally no difference, but makes editing of the statements easier. Since reference water levels are not deleted all at once before (re-)importing anymore, take the chance to report those that were deleted.
author Tom Gottfried <tom@intevation.de>
date Mon, 06 May 2019 13:25:49 +0200
parents 6a44a89ffb51
children a5448426e4e2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
3 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
6 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
10 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
11 // Author(s):
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
14
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
15 package imports
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
16
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
17 import (
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
18 "context"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
19 "database/sql"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
20 "fmt"
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
21 "log"
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
22 "strings"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
23
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
24 "gemma.intevation.de/gemma/pkg/soap"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/soap/erdms"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
26 )
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
27
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
28 const (
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
29 selectUserCountriesSQL = `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
30 SELECT DISTINCT country FROM users.list_users
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
31 WHERE country <> '--'
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
32 `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
33 )
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
34
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
35 func userCountries(ctx context.Context, conn *sql.Conn) ([]string, error) {
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
36 rows, err := conn.QueryContext(ctx, selectUserCountriesSQL)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
37 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
38 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
39 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
40 defer rows.Close()
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
41
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
42 var countries []string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
43
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
44 for rows.Next() {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
45 var country string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
46 if err = rows.Scan(&country); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
47 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
48 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
49 countries = append(countries, country)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
50 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
51
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
52 if err := rows.Err(); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
53 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
54 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
55
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
56 return countries, nil
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
57 }
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
58
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
59 func getRisData(
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
60 ctx context.Context,
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
61 conn *sql.Conn,
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
62 feedback Feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
63 username string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
64 password string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
65 URL string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
66 insecure bool,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
67 funcode string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
68 ) ([]*erdms.GetRisDataXMLResponse, error) {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
69
3163
d9903cb34842 Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents: 3161
diff changeset
70 countries, err := userCountries(ctx, conn)
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
71 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
72 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
73 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
74
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
75 var auth *soap.BasicAuth
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
76 if username != "" {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
77 auth = &soap.BasicAuth{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
78 Login: username,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
79 Password: password,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
80 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
81 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
82
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
83 client := erdms.NewRefService(URL, insecure, auth)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
84
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
85 var responseData []*erdms.GetRisDataXMLResponse
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
86 for _, country := range countries {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
87
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
88 feedback.Info("Request RIS index for country %s", country)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
89
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
90 request := &erdms.GetRisDataXML{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
91 GetRisDataXMLType: &erdms.GetRisDataXMLType{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
92 Subcode: erdms.NoNS{Text: country + "%"},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
93 Funcode: erdms.NoNS{Text: funcode},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
94 },
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
95 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
96
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
97 const maxTries = 3
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
98
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
99 tries := 0
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
100
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
101 again:
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
102 data, err := client.GetRisDataXML(request)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
103 if err != nil {
3161
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
104 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
105 log.Println("warn: ERDMS SOAP request timed out. Trying again.")
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
106 tries++
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
107 goto again
6a44a89ffb51 SOAP: Added a globally configurable timeout (default 1min) till a SOAP request is canceled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2988
diff changeset
108 }
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
109 return nil, fmt.Errorf("Error requesting ERDMS service: %v", err)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
110 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
111 responseData = append(responseData, data)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
112 }
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
113
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
114 feedback.Info("Import data for countries: %s.",
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
115 strings.Join(countries, ", "))
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
116
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
117 return responseData, nil
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
118 }