Mercurial > gemma

annotate pkg/wfs/download.go @ 2672:b997e1fd1d3d import-overview-rework

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed warning SQL prefix for selection.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Thu, 14 Mar 2019 17:29:36 +0100
parents 02505fcff63c
children d7ef169fd0d3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package wfs
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "bufio"
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
18 "bytes"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "encoding/xml"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "errors"
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
21 "fmt"
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
22 "io"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "log"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "net/http"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "net/url"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "strconv"
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
27 "strings"
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "golang.org/x/net/html/charset"
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32 var (
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
33 // ErrNoSuchFeatureType is returned when a feature is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
34 ErrNoSuchFeatureType = errors.New("No such feature type")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
35 // ErrGetFeatureNotSupported is returned when GetFeature is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
36 ErrGetFeatureNotSupported = errors.New("GetFeature not supported")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
37 // ErrMethodGetNotSupported is returned when the GET is not supported.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
38 ErrMethodGetNotSupported = errors.New("GET not supported")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
39 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted.
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
40 ErrNoNumberMatchedFound = errors.New("No numberMatched attribute found")
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
41 // ErrOutputFormatNotSupported is returned if a output format is
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
42 // not supported.
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
43 ErrOutputFormatNotSupported = errors.New("Output format not supported")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 )
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45
2622
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
46 var FormatGeoJSON = []string{"geojson", "application/json"}
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
47
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
48 // GetCapabilities downloads a capabilities document for a given URL.
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49 func GetCapabilities(capURL string) (*Capabilities, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 base, err := url.Parse(capURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 v.Set("REQUEST", "GetCapabilities")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 base.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 baseURL := base.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 resp, err := http.Get(baseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 return nil, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 caps, err := ParseCapabilities(bufio.NewReader(resp.Body))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68 if err == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 caps.BaseURL = baseURL
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71 return caps, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
74 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
75
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79 v.Set("resultType", "hits")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80 v.Set("VERSION", version)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81 v.Set("TYPENAMES", featureType)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
82
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
83 q := *u
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
84 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
86 resp, err := http.Get(q.String())
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
87 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
88 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
89 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
90 defer resp.Body.Close()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
91 dec := xml.NewDecoder(resp.Body)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92 dec.CharsetReader = charset.NewReaderLabel
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 var result struct {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95 NumberMatched *int `xml:"numberMatched,attr"`
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
97
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
98 if err := dec.Decode(&result); err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
99 return 0, err
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
100 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
101
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
102 if result.NumberMatched == nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
103 return 0, ErrNoNumberMatchedFound
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
104 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
105
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
106 return *result.NumberMatched, nil
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
107 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
108
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
109 // GetFeaturesGET constructs a list of URLs to get features
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
110 // for a given feature type from a WFS servers.
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
111 func GetFeaturesGET(
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
112 caps *Capabilities,
2622
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
113 featureTypeName string,
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
114 outputFormats []string,
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
115 sortBy string,
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
116 ) ([]string, error) {
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
117
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
118 feature := caps.FindFeatureType(featureTypeName)
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
119 if feature == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
120 return nil, ErrNoSuchFeatureType
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
121 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
122 op := caps.FindOperation("GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
123 if op == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
124 return nil, ErrGetFeatureNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
125 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
126
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
127 if op.DCP.HTTP.Get == nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
128 return nil, ErrMethodGetNotSupported
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
129 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
130
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
131 getRaw := op.DCP.HTTP.Get.HRef
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
132 getU, err := url.Parse(getRaw)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
133 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
134 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
135 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
136 // The URL could be relative so resolve against Capabilities URL.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
137 if !getU.IsAbs() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
138 base, err := url.Parse(caps.BaseURL)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
139 if err != nil {
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
140 return nil, err
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
141 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
142 getU = getU.ResolveReference(base)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
143 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
144
2622
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
145 var outputFormat string
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
146
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
147 if len(outputFormats) > 0 {
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
148 if outputFormat = op.SupportsOutputFormat(outputFormats...); outputFormat == "" {
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
149 return nil, ErrOutputFormatNotSupported
02505fcff63c WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2620
diff changeset
150 }
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
151 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
152
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
153 wfsVersion := caps.HighestWFSVersion(WFS200)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
154
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
155 featuresPerPage, supportsPaging := op.FeaturesPerPage()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
156
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157 var numFeatures int
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
158
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 if supportsPaging {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
160 log.Printf("info: Paging supported with %d feature per page.\n",
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
161 featuresPerPage)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 if !op.SupportsHits() {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164 supportsPaging = false
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 if err != nil {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168 log.Printf("error: %v\n", err)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 supportsPaging = false
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 } else {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
171 log.Printf("info: Number of features: %d\n", numFeatures)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 var downloadURLs []string
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
177 wfs2 := !versionIsLess(wfsVersion, WFS200)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
178
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
179 addNS := func(v url.Values) {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
180 if len(feature.Namespaces) == 0 {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
181 return
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
182 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
183 // Only use first namespace
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
184 ns := feature.Namespaces[0]
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
185 if wfs2 {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
186 v.Set("NAMESPACES", fmt.Sprintf("(%s,%s)", ns.Space, ns.Local))
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
187 } else {
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
188 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local))
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
189 }
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
190 }
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
191
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
192 addOutputFormat := func(v url.Values) {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
193 if outputFormat != "" {
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
194 v.Set("outputFormat", outputFormat)
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
195 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
196 }
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
197
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
198 addSortBy := func(v url.Values) {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
199 if sortBy != "" {
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
200 v.Set("sortBy", sortBy)
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
201 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
202 }
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
203
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
204 if supportsPaging {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205 pagedURL := func(ofs, count int) string {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 v.Set("startIndex", strconv.Itoa(ofs))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 if wfs2 {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 v.Set("count", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 v.Set("maxFeatures", strconv.Itoa(count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
217 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
218 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
219 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
220 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222 return q.String()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
224 if numFeatures <= featuresPerPage {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
225 log.Println("info: All features can be fetched in one page.")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
226 downloadURLs = []string{pagedURL(0, numFeatures)}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 } else {
2096
8b7dee291488 WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1935
diff changeset
228 log.Println("info: Features need to be downloaded in pages.")
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
229 for pos := 0; pos < numFeatures; {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
230 var count int
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
231 if rest := numFeatures - pos; rest >= numFeatures {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232 count = numFeatures
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
233 } else {
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
234 count = rest
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
235 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
236 downloadURLs = append(downloadURLs, pagedURL(pos, count))
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
237 pos += count
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
238 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
239 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
240 } else { // No paging support.
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
241 v := url.Values{}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
242 v.Set("SERVICE", "WFS")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
243 v.Set("REQUEST", "GetFeature")
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
244 v.Set("VERSION", wfsVersion)
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
245 v.Set("TYPENAMES", featureTypeName)
1614
efc409e330a6 WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1608
diff changeset
246 addNS(v)
1616
f59550310143 WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1614
diff changeset
247 addOutputFormat(v)
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
248 addSortBy(v)
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
249 q := *getU
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
250 q.RawQuery = v.Encode()
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
251 downloadURLs = []string{q.String()}
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
252 }
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
253
1619
3093bab05c81 WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1616
diff changeset
254 return downloadURLs, nil
1608
427f9010b4a9 WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
255 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
256
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2096
diff changeset
257 func downloadURL(url string, handler func(string, io.Reader) error) error {
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
258 resp, err := http.Get(url)
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
259 if err != nil {
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
260 return err
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
261 }
1626
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
262 if resp.StatusCode < 200 || resp.StatusCode > 299 {
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
263 return fmt.Errorf("Invalid HTTP status code: %d (%s)",
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
264 resp.StatusCode, resp.Status)
92da44ba610c WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1624
diff changeset
265 }
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
266 defer resp.Body.Close()
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
267
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
268 var already bytes.Buffer
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
269
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
270 // Prevent the XML reader from consuming everything.
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
271 limit := io.LimitReader(resp.Body, 16*1024)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
272 in := io.TeeReader(limit, &already)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
273
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
274 if err := scanExceptionReport(in); err != nil {
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
275 return err
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
276 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
277
2110
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
278 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body)
fc4ebe7eaeff WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2109
diff changeset
279 return handler(url, multi)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
280 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
281
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
282 type ExceptionReport struct {
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
283 Code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
284 Text string
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
285 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
286
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
287 func (er *ExceptionReport) Error() string {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
288 return fmt.Sprintf(
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
289 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s",
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
290 er.Code, er.Text)
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
291 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
292
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
293 func scanExceptionReport(r io.Reader) *ExceptionReport {
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
294
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
295 const ows = "http://www.opengis.net/ows/1.1"
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
296
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
297 decoder := xml.NewDecoder(r)
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
298 decoder.CharsetReader = charset.NewReaderLabel
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
299
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
300 var (
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
301 isError bool
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
302 code string
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
303 depth int
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
304 text strings.Builder
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
305 )
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
306
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
307 type tokenFunc func(xml.Token) tokenFunc
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
308
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
309 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
310
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
311 exceptionReportFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
312 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
313 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
314 return exceptionReportFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
315 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
316 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
317 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
318 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
319 isError = true
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
320 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
321 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
322
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
323 exceptionFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
324 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
325 if !ok {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
326 return exceptionFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
327 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
328 if e.Name.Local == "Exception" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
329 for i := range e.Attr {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
330 at := &e.Attr[i]
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
331 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
332 code = at.Value
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
333 break
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
334 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
335 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
336 return exceptionTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
337 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
338 return exceptionFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
339 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
340
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
341 exceptionTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
342 e, ok := t.(xml.StartElement)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
343 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
344 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
345 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
346 return exceptionTextFn
2109
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
347 }
86c88fc0ff5e WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2106
diff changeset
348
2118
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
349 collectTextFn = func(t xml.Token) tokenFunc {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
350 switch e := t.(type) {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
351 case xml.StartElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
352 depth++
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
353 case xml.CharData:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
354 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
355 text.Write(e)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
356 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
357 case xml.EndElement:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
358 if depth == 0 {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
359 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
360 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
361 depth--
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
362 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
363 return collectTextFn
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
364 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
365
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
366 tokens:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
367 for fn := exceptionReportFn; fn != nil; {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
368 tok, err := decoder.Token()
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
369 switch {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
370 case tok == nil && err == io.EOF:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
371 break tokens
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
372 case err != nil:
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
373 return nil
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
374 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
375 fn = fn(tok)
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
376 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
377
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
378 if isError {
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
379 return &ExceptionReport{
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
380 Code: code,
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
381 Text: text.String(),
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
382 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
383 }
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
384
8a6ed058af22 WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2110
diff changeset
385 return nil
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
386 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
387
1689
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
388 // DownloadURLs does the actual GetFeature requests downloads
6caf5cd6249e WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1626
diff changeset
389 // and hands the resulting io.Readers over to the given handler.
2106
2b72f5e005aa WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2096
diff changeset
390 func DownloadURLs(urls []string, handler func(string, io.Reader) error) error {
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
391 for _, url := range urls {
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
392 if err := downloadURL(url, handler); err != nil {
1935
73f9e15d0384 WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1689
diff changeset
393 return err
1624
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
394 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
395 }
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
396 return nil
943823d03d50 WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1619
diff changeset
397 }