Mercurial > gemma

annotate pkg/wfs/download.go @ 4990:16259efa828f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Dont try to download from WFS if they dont have features.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Thu, 05 Mar 2020 17:56:08 +0100
parents ed0de319d09a
children 5f47eeea988d
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package wfs
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "bufio"
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
18 "bytes"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "encoding/xml"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "errors"
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
21 "fmt"
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
22 "io"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "log"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "net/http"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "net/url"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "strconv"
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
27 "strings"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "golang.org/x/net/html/charset"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32 var (
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
33 // ErrNoSuchFeatureType is returned when a feature is not supported.
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
34 ErrNoSuchFeatureType = errors.New("no such feature type")
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
35 // ErrGetFeatureNotSupported is returned when GetFeature is not supported.
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
36 ErrGetFeatureNotSupported = errors.New("method GetFeature not supported")
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
37 // ErrMethodGetNotSupported is returned when the GET is not supported.
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
38 ErrMethodGetNotSupported = errors.New("method GET not supported")
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
39 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted.
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
40 ErrNoNumberMatchedFound = errors.New("no numberMatched attribute found")
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
41 // ErrOutputFormatNotSupported is returned if a output format is
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
42 // not supported.
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
43 ErrOutputFormatNotSupported = errors.New("output format not supported")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
46 // GetCapabilities downloads a capabilities document for a given URL.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 func GetCapabilities(capURL string) (*Capabilities, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49 base, err := url.Parse(capURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 v.Set("REQUEST", "GetCapabilities")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 base.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 baseURL := base.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 resp, err := http.Get(baseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 caps, err := ParseCapabilities(bufio.NewReader(resp.Body))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 if err == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 caps.BaseURL = baseURL
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 return caps, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
74 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
75 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77 v.Set("resultType", "hits")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 v.Set("VERSION", version)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79 v.Set("TYPENAMES", featureType)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81 q := *u
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
82 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
83
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
84 resp, err := http.Get(q.String())
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
86 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
87 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
88 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
89 dec := xml.NewDecoder(resp.Body)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
90 dec.CharsetReader = charset.NewReaderLabel
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
91
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92 var result struct {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93 NumberMatched *int `xml:"numberMatched,attr"`
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96 if err := dec.Decode(&result); err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
97 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
98 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
99
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
100 if result.NumberMatched == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
101 return 0, ErrNoNumberMatchedFound
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
102 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
103
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
104 return *result.NumberMatched, nil
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
105 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
106
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
107 // GetFeaturesGET constructs a list of URLs to get features
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
108 // for a given feature type from a WFS servers.
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
109 func GetFeaturesGET(
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
110 caps *Capabilities,
2622
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
111 featureTypeName string,
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
112 outputFormats []string,
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
113 sortBy string,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
114 ) ([]string, error) {
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
115
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
116 feature := caps.FindFeatureType(featureTypeName)
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
117 if feature == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
118 return nil, ErrNoSuchFeatureType
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
119 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
120 op := caps.FindOperation("GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
121 if op == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
122 return nil, ErrGetFeatureNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
123 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
124
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
125 if op.DCP.HTTP.Get == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
126 return nil, ErrMethodGetNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
127 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
128
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
129 getRaw := op.DCP.HTTP.Get.HRef
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
130 getU, err := url.Parse(getRaw)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
131 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
132 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
133 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
134 // The URL could be relative so resolve against Capabilities URL.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
135 if !getU.IsAbs() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
136 base, err := url.Parse(caps.BaseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
137 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
138 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
139 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
140 getU = getU.ResolveReference(base)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
141 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
142
2622
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
143 var outputFormat string
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
144
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
145 if len(outputFormats) > 0 {
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
146 if outputFormat = op.SupportsOutputFormat(outputFormats...); outputFormat == "" {
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
147 return nil, ErrOutputFormatNotSupported
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
148 }
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
149 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
150
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
151 wfsVersion := caps.HighestWFSVersion(WFS200)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
152
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
153 featuresPerPage, supportsPaging := op.FeaturesPerPage()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
154
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
155 var numFeatures int
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
156
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157 if supportsPaging {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
158 log.Printf("info: Paging supported with %d feature per page.\n",
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 featuresPerPage)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
160
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
161 if !op.SupportsHits() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162 supportsPaging = false
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 log.Printf("error: %v\n", err)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 supportsPaging = false
4990
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
168 } else if numFeatures == 0 {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
169 return nil, nil
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 } else {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
171 log.Printf("info: Number of features: %d\n", numFeatures)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 var downloadURLs []string
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
177 wfs2 := !versionIsLess(wfsVersion, WFS200)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
178
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
179 addNS := func(v url.Values) {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
180 if len(feature.Namespaces) == 0 {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
181 return
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
182 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
183 // Only use first namespace
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
184 ns := feature.Namespaces[0]
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
185 if wfs2 {
4880
ed0de319d09a Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents: 4166
diff changeset
186 v.Set(
ed0de319d09a Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents: 4166
diff changeset
187 "NAMESPACES", fmt.Sprintf("xmlns(%s,%s)", ns.Space, ns.Local))
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
188 } else {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
189 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local))
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
190 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
191 }
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
192
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
193 addOutputFormat := func(v url.Values) {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
194 if outputFormat != "" {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
195 v.Set("outputFormat", outputFormat)
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
196 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
197 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
198
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
199 addSortBy := func(v url.Values) {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
200 if sortBy != "" {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
201 v.Set("sortBy", sortBy)
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
202 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
203 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
204
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205 if supportsPaging {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 pagedURL := func(ofs, count int) string {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 v.Set("startIndex", strconv.Itoa(ofs))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 if wfs2 {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213 v.Set("count", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 v.Set("maxFeatures", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
218 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
219 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
220 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223 return q.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
224 }
4990
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
225 if numFeatures > 0 {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
226 if numFeatures <= featuresPerPage {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
227 log.Println("info: All features can be fetched in one page.")
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
228 downloadURLs = []string{pagedURL(0, numFeatures)}
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
229 } else {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
230 log.Println("info: Features need to be downloaded in pages.")
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
231 for pos := 0; pos < numFeatures; {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
232 var count int
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
233 if rest := numFeatures - pos; rest >= numFeatures {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
234 count = numFeatures
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
235 } else {
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
236 count = rest
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
237 }
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
238 downloadURLs = append(downloadURLs, pagedURL(pos, count))
16259efa828f Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4880
diff changeset
239 pos += count
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
240 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
241 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
242 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
243 } else { // No paging support.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
244 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
245 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
246 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
247 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
248 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
249 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
250 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
251 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
252 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
253 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
254 downloadURLs = []string{q.String()}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
255 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
256
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
257 return downloadURLs, nil
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
258 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
259
2719
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
260 func downloadURL(user, password, url string, handler func(string, io.Reader) error) error {
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
261 req, err := http.NewRequest(http.MethodGet, url, nil)
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
262 if err != nil {
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
263 return err
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
264 }
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
265
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
266 if user != "" || password != "" {
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
267 req.SetBasicAuth(user, password)
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
268 }
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
269
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
270 resp, err := http.DefaultClient.Do(req)
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
271
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
272 if err != nil {
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
273 return err
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
274 }
1626
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
275 if resp.StatusCode < 200 || resp.StatusCode > 299 {
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
276 return fmt.Errorf("invalid HTTP status code: %d (%s)",
1626
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
277 resp.StatusCode, resp.Status)
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
278 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
279 defer resp.Body.Close()
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
280
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
281 var already bytes.Buffer
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
282
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
283 // Prevent the XML reader from consuming everything.
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
284 limit := io.LimitReader(resp.Body, 16*1024)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
285 in := io.TeeReader(limit, &already)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
286
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
287 if err := scanExceptionReport(in); err != nil {
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
288 return err
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
289 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
290
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
291 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
292 return handler(url, multi)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
293 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
294
4166
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
295 // ExceptionReport is an error with the extract code and
04876d865528 Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2719
diff changeset
296 // text from an OWS exception document.
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
297 type ExceptionReport struct {
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
298 Code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
299 Text string
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
300 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
301
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
302 func (er *ExceptionReport) Error() string {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
303 return fmt.Sprintf(
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
304 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s",
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
305 er.Code, er.Text)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
306 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
307
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
308 func scanExceptionReport(r io.Reader) *ExceptionReport {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
309
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
310 const ows = "http://www.opengis.net/ows/1.1"
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
311
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
312 decoder := xml.NewDecoder(r)
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
313 decoder.CharsetReader = charset.NewReaderLabel
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
314
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
315 var (
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
316 isError bool
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
317 code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
318 depth int
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
319 text strings.Builder
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
320 )
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
321
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
322 type tokenFunc func(xml.Token) tokenFunc
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
323
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
324 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
325
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
326 exceptionReportFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
327 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
328 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
329 return exceptionReportFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
330 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
331 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
332 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
333 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
334 isError = true
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
335 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
336 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
337
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
338 exceptionFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
339 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
340 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
341 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
342 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
343 if e.Name.Local == "Exception" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
344 for i := range e.Attr {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
345 at := &e.Attr[i]
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
346 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
347 code = at.Value
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
348 break
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
349 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
350 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
351 return exceptionTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
352 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
353 return exceptionFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
354 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
355
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
356 exceptionTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
357 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
358 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
359 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
360 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
361 return exceptionTextFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
362 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
363
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
364 collectTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
365 switch e := t.(type) {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
366 case xml.StartElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
367 depth++
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
368 case xml.CharData:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
369 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
370 text.Write(e)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
371 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
372 case xml.EndElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
373 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
374 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
375 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
376 depth--
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
377 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
378 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
379 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
380
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
381 tokens:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
382 for fn := exceptionReportFn; fn != nil; {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
383 tok, err := decoder.Token()
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
384 switch {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
385 case tok == nil && err == io.EOF:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
386 break tokens
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
387 case err != nil:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
388 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
389 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
390 fn = fn(tok)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
391 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
392
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
393 if isError {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
394 return &ExceptionReport{
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
395 Code: code,
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
396 Text: text.String(),
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
397 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
398 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
399
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
400 return nil
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
401 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
402
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
403 // DownloadURLs does the actual GetFeature requests downloads
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
404 // and hands the resulting io.Readers over to the given handler.
2719
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
405 func DownloadURLs(user, password string, urls []string, handler func(string, io.Reader) error) error {
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
406 for _, url := range urls {
2719
2b6f73c30016 WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2702
diff changeset
407 if err := downloadURL(user, password, url, handler); err != nil {
1935
73f9e15d0384 WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1689
diff changeset
408 return err
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
409 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
410 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
411 return nil
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
412 }