Mercurial > gemma

annotate pkg/wfs/download.go @ 2549:9bf6b767a56a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
client: refactored and improved splitscreen for diagrams To make different diagrams possible, the splitscreen view needed to be decoupled from the cross profiles. Also the style has changed to make it more consistent with the rest of the app. The standard box header is now used and there are collapse and expand animations.
author Markus Kottlaender <markus@intevation.de>
date Fri, 08 Mar 2019 08:50:47 +0100
parents 8a6ed058af22
children a8ebc785823d
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package wfs
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "bufio"
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
18 "bytes"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "encoding/xml"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "errors"
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
21 "fmt"
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
22 "io"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "log"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "net/http"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "net/url"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "strconv"
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
27 "strings"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
29 "gemma.intevation.de/gemma/pkg/config"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "golang.org/x/net/html/charset"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 var (
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
34 // ErrNoSuchFeatureType is returned when a feature is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
35 ErrNoSuchFeatureType = errors.New("No such feature type")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
36 // ErrGetFeatureNotSupported is returned when GetFeature is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
37 ErrGetFeatureNotSupported = errors.New("GetFeature not supported")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
38 // ErrMethodGetNotSupported is returned when the GET is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
39 ErrMethodGetNotSupported = errors.New("GET not supported")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
40 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
41 ErrNoNumberMatchedFound = errors.New("No numberMatched attribute found")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
42 // ErrOutputFormatNotSupported is returned if a output format is
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
43 // not supported.
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
44 ErrOutputFormatNotSupported = errors.New("Output format not supported")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
47 // GetCapabilities downloads a capabilities document for a given URL.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48 func GetCapabilities(capURL string) (*Capabilities, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 base, err := url.Parse(capURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 v.Set("REQUEST", "GetCapabilities")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 base.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 baseURL := base.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 resp, err := http.Get(baseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 caps, err := ParseCapabilities(bufio.NewReader(resp.Body))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 if err == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68 caps.BaseURL = baseURL
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 return caps, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
74
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
75 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 v.Set("resultType", "hits")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79 v.Set("VERSION", version)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80 v.Set("TYPENAMES", featureType)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
82 q := *u
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
83 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
84
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85 resp, err := http.Get(q.String())
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
86 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
87 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
88 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
89 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
90 dec := xml.NewDecoder(resp.Body)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
91 dec.CharsetReader = charset.NewReaderLabel
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93 var result struct {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 NumberMatched *int `xml:"numberMatched,attr"`
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
97 if err := dec.Decode(&result); err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
98 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
99 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
100
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
101 if result.NumberMatched == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
102 return 0, ErrNoNumberMatchedFound
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
103 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
104
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
105 return *result.NumberMatched, nil
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
106 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
107
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
108 // GetFeaturesGET constructs a list of URLs to get features
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
109 // for a given feature type from a WFS servers.
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
110 func GetFeaturesGET(
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
111 caps *Capabilities,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
112 featureTypeName,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
113 outputFormat string,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
114 sortBy string,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
115 ) ([]string, error) {
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
116
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
117 feature := caps.FindFeatureType(featureTypeName)
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
118 if feature == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
119 return nil, ErrNoSuchFeatureType
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
120 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
121 op := caps.FindOperation("GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
122 if op == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
123 return nil, ErrGetFeatureNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
124 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
125
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
126 if op.DCP.HTTP.Get == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
127 return nil, ErrMethodGetNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
128 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
129
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
130 getRaw := op.DCP.HTTP.Get.HRef
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
131 getU, err := url.Parse(getRaw)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
132 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
133 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
134 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
135 // The URL could be relative so resolve against Capabilities URL.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
136 if !getU.IsAbs() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
137 base, err := url.Parse(caps.BaseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
138 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
139 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
140 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
141 getU = getU.ResolveReference(base)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
142 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
143
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
144 if !op.SupportsOutputFormat(outputFormat) {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
145 return nil, ErrOutputFormatNotSupported
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
146 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
147
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
148 wfsVersion := caps.HighestWFSVersion(WFS200)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
149
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
150 featuresPerPage, supportsPaging := op.FeaturesPerPage()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
151
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
152 var numFeatures int
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
153
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
154 if supportsPaging {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
155 log.Printf("info: Paging supported with %d feature per page.\n",
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
156 featuresPerPage)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
158 if !op.SupportsHits() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 supportsPaging = false
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
160 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
161 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 log.Printf("error: %v\n", err)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164 supportsPaging = false
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 } else {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
166 log.Printf("info: Number of features: %d\n", numFeatures)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
171 var downloadURLs []string
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
172 wfs2 := !versionIsLess(wfsVersion, WFS200)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
173
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
174 addNS := func(v url.Values) {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
175 if len(feature.Namespaces) == 0 {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
176 return
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
177 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
178 // Only use first namespace
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
179 ns := feature.Namespaces[0]
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
180 if wfs2 {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
181 v.Set("NAMESPACES", fmt.Sprintf("(%s,%s)", ns.Space, ns.Local))
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
182 } else {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
183 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local))
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
184 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
185 }
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
186
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
187 addOutputFormat := func(v url.Values) {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
188 if outputFormat != "" {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
189 v.Set("outputFormat", outputFormat)
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
190 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
191 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
192
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
193 addSortBy := func(v url.Values) {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
194 if sortBy != "" {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
195 v.Set("sortBy", sortBy)
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
196 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
197 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
198
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
199 if supportsPaging {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
200 pagedURL := func(ofs, count int) string {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
201 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
202 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
203 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
204 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205 v.Set("startIndex", strconv.Itoa(ofs))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 if wfs2 {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 v.Set("count", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 v.Set("maxFeatures", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
212 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
213 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
214 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217 return q.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
218 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 if numFeatures <= featuresPerPage {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
220 log.Println("info: All features can be fetched in one page.")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 downloadURLs = []string{pagedURL(0, numFeatures)}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222 } else {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
223 log.Println("info: Features need to be downloaded in pages.")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
224 for pos := 0; pos < numFeatures; {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
225 var count int
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
226 if rest := numFeatures - pos; rest >= numFeatures {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 count = numFeatures
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
229 count = rest
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
230 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
231 downloadURLs = append(downloadURLs, pagedURL(pos, count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232 pos += count
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
233 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
234 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
235 } else { // No paging support.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
236 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
237 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
238 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
239 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
240 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
241 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
242 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
243 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
244 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
245 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
246 downloadURLs = []string{q.String()}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
247 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
248
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
249 return downloadURLs, nil
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
250 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
251
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2096
diff changeset
252 func downloadURL(url string, handler func(string, io.Reader) error) error {
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
253 resp, err := http.Get(url)
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
254 if err != nil {
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
255 return err
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
256 }
1626
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
257 if resp.StatusCode < 200 || resp.StatusCode > 299 {
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
258 return fmt.Errorf("Invalid HTTP status code: %d (%s)",
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
259 resp.StatusCode, resp.Status)
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
260 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
261 defer resp.Body.Close()
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
262
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
263 var already bytes.Buffer
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
264
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
265 // Prevent the XML reader from consuming everything.
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
266 limit := io.LimitReader(resp.Body, 16*1024)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
267 in := io.TeeReader(limit, &already)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
268
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
269 if err := scanExceptionReport(in); err != nil {
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
270 return err
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
271 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
272
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
273 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
274 return handler(url, multi)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
275 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
276
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
277 type ExceptionReport struct {
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
278 Code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
279 Text string
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
280 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
281
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
282 func (er *ExceptionReport) Error() string {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
283 return fmt.Sprintf(
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
284 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s",
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
285 er.Code, er.Text)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
286 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
287
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
288 func scanExceptionReport(r io.Reader) *ExceptionReport {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
289
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
290 const ows = "http://www.opengis.net/ows/1.1"
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
291
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
292 decoder := xml.NewDecoder(r)
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
293 decoder.CharsetReader = charset.NewReaderLabel
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
294
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
295 var (
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
296 isError bool
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
297 code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
298 depth int
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
299 text strings.Builder
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
300 )
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
301
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
302 type tokenFunc func(xml.Token) tokenFunc
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
303
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
304 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
305
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
306 exceptionReportFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
307 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
308 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
309 return exceptionReportFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
310 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
311 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
312 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
313 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
314 isError = true
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
315 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
316 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
317
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
318 exceptionFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
319 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
320 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
321 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
322 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
323 if e.Name.Local == "Exception" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
324 for i := range e.Attr {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
325 at := &e.Attr[i]
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
326 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
327 code = at.Value
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
328 break
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
329 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
330 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
331 return exceptionTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
332 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
333 return exceptionFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
334 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
335
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
336 exceptionTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
337 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
338 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
339 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
340 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
341 return exceptionTextFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
342 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
343
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
344 collectTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
345 switch e := t.(type) {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
346 case xml.StartElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
347 depth++
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
348 case xml.CharData:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
349 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
350 text.Write(e)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
351 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
352 case xml.EndElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
353 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
354 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
355 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
356 depth--
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
357 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
358 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
359 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
360
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
361 tokens:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
362 for fn := exceptionReportFn; fn != nil; {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
363 tok, err := decoder.Token()
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
364 switch {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
365 case tok == nil && err == io.EOF:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
366 break tokens
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
367 case err != nil:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
368 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
369 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
370 fn = fn(tok)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
371 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
372
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
373 if isError {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
374 return &ExceptionReport{
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
375 Code: code,
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
376 Text: text.String(),
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
377 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
378 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
379
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
380 return nil
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
381 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
382
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
383 // DownloadURLs does the actual GetFeature requests downloads
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
384 // and hands the resulting io.Readers over to the given handler.
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2096
diff changeset
385 func DownloadURLs(urls []string, handler func(string, io.Reader) error) error {
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
386 config.WaitReady()
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
387 for _, url := range urls {
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
388 if err := downloadURL(url, handler); err != nil {
1935
73f9e15d0384 WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1689
diff changeset
389 return err
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
390 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
391 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
392 return nil
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
393 }