Mercurial > gemma

annotate pkg/imports/erdms.go @ 2988:e1ccc8438529

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Wed, 10 Apr 2019 11:00:23 +0200
parents eb1d119f253f
children 6a44a89ffb51
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
3 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
6 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
10 //
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
11 // Author(s):
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
14
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
15 package imports
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
16
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
17 import (
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
18 "context"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
19 "database/sql"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
20 "fmt"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
21 "strings"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
22
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
23 "gemma.intevation.de/gemma/pkg/soap"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
24 "gemma.intevation.de/gemma/pkg/soap/erdms"
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
25 )
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
26
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
27 const (
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
28 selectUserCountriesSQL = `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
29 SELECT DISTINCT country FROM users.list_users
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
30 WHERE country <> '--'
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
31 `
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
32 )
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
33
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
34 func userCountries(ctx context.Context, tx *sql.Tx) ([]string, error) {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
35 rows, err := tx.QueryContext(ctx, selectUserCountriesSQL)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
36 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
37 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
38 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
39 defer rows.Close()
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
40
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
41 var countries []string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
42
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
43 for rows.Next() {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
44 var country string
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
45 if err = rows.Scan(&country); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
46 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
47 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
48 countries = append(countries, country)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
49 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
50
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
51 if err := rows.Err(); err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
52 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
53 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
54
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
55 return countries, nil
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
56 }
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
57
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
58 func getRisData(
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
59 tx *sql.Tx,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
60 ctx context.Context,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
61 feedback Feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
62 username string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
63 password string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
64 URL string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
65 insecure bool,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
66 funcode string,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
67 ) ([]*erdms.GetRisDataXMLResponse, error) {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
68
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
69 countries, err := userCountries(ctx, tx)
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
70 if err != nil {
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
71 return nil, err
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
72 }
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
73
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
74 var auth *soap.BasicAuth
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
75 if username != "" {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
76 auth = &soap.BasicAuth{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
77 Login: username,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
78 Password: password,
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
79 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
80 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
81
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
82 client := erdms.NewRefService(URL, insecure, auth)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
83
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
84 var responseData []*erdms.GetRisDataXMLResponse
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
85 for _, country := range countries {
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
86
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
87 feedback.Info("Request RIS index for country %s", country)
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
88
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
89 request := &erdms.GetRisDataXML{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
90 GetRisDataXMLType: &erdms.GetRisDataXMLType{
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
91 Subcode: erdms.NoNS{Text: country + "%"},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
92 Funcode: erdms.NoNS{Text: funcode},
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
93 },
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
94 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
95
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
96 data, err := client.GetRisDataXML(request)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
97 if err != nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
98 return nil, fmt.Errorf("Error requesting ERDMS service: %v", err)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
99 }
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
100 responseData = append(responseData, data)
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
101 }
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
102
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
103 feedback.Info("Import data for countries: %s.",
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
104 strings.Join(countries, ", "))
2988
e1ccc8438529 DMV: separated fetching list of countries from database from requesting the RIS index. Also added some logging and better error handling.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2633
diff changeset
105
2633
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
106 return responseData, nil
eb1d119f253f Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
diff changeset
107 }