Mercurial > gemma
annotate pkg/wfs/download.go @ 2650:a308baa7e7af
Merged
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Thu, 14 Mar 2019 14:50:20 +0100 |
parents | 02505fcff63c |
children | d7ef169fd0d3 |
rev | line source |
---|---|
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package wfs |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
18 "bytes" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "encoding/xml" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "errors" |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
21 "fmt" |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
22 "io" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "log" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "net/http" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "net/url" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 "strconv" |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
27 "strings" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "golang.org/x/net/html/charset" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 var ( |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
33 // ErrNoSuchFeatureType is returned when a feature is not supported. |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
34 ErrNoSuchFeatureType = errors.New("No such feature type") |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
35 // ErrGetFeatureNotSupported is returned when GetFeature is not supported. |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
36 ErrGetFeatureNotSupported = errors.New("GetFeature not supported") |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
37 // ErrMethodGetNotSupported is returned when the GET is not supported. |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
38 ErrMethodGetNotSupported = errors.New("GET not supported") |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
39 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted. |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
40 ErrNoNumberMatchedFound = errors.New("No numberMatched attribute found") |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
41 // ErrOutputFormatNotSupported is returned if a output format is |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
42 // not supported. |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
43 ErrOutputFormatNotSupported = errors.New("Output format not supported") |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
46 var FormatGeoJSON = []string{"geojson", "application/json"} |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
47 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
48 // GetCapabilities downloads a capabilities document for a given URL. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 func GetCapabilities(capURL string) (*Capabilities, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 base, err := url.Parse(capURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 v.Set("REQUEST", "GetCapabilities") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 base.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 baseURL := base.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 resp, err := http.Get(baseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 caps, err := ParseCapabilities(bufio.NewReader(resp.Body)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 if err == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 caps.BaseURL = baseURL |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 return caps, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 v.Set("resultType", "hits") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 v.Set("VERSION", version) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 v.Set("TYPENAMES", featureType) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 q := *u |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 resp, err := http.Get(q.String()) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 dec := xml.NewDecoder(resp.Body) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 dec.CharsetReader = charset.NewReaderLabel |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 var result struct { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 NumberMatched *int `xml:"numberMatched,attr"` |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 if err := dec.Decode(&result); err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 if result.NumberMatched == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 return 0, ErrNoNumberMatchedFound |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 return *result.NumberMatched, nil |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
108 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
109 // GetFeaturesGET constructs a list of URLs to get features |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
110 // for a given feature type from a WFS servers. |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
111 func GetFeaturesGET( |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
112 caps *Capabilities, |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
113 featureTypeName string, |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
114 outputFormats []string, |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
115 sortBy string, |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
116 ) ([]string, error) { |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
117 |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
118 feature := caps.FindFeatureType(featureTypeName) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
119 if feature == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
120 return nil, ErrNoSuchFeatureType |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
121 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
122 op := caps.FindOperation("GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
123 if op == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
124 return nil, ErrGetFeatureNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
125 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
126 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
127 if op.DCP.HTTP.Get == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
128 return nil, ErrMethodGetNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
129 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
130 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
131 getRaw := op.DCP.HTTP.Get.HRef |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
132 getU, err := url.Parse(getRaw) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
133 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
134 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
135 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
136 // The URL could be relative so resolve against Capabilities URL. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
137 if !getU.IsAbs() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
138 base, err := url.Parse(caps.BaseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
139 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
140 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
141 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
142 getU = getU.ResolveReference(base) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
144 |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
145 var outputFormat string |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
146 |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
147 if len(outputFormats) > 0 { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
148 if outputFormat = op.SupportsOutputFormat(outputFormats...); outputFormat == "" { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
149 return nil, ErrOutputFormatNotSupported |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
150 } |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
151 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
152 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
153 wfsVersion := caps.HighestWFSVersion(WFS200) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 featuresPerPage, supportsPaging := op.FeaturesPerPage() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 var numFeatures int |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 if supportsPaging { |
2096
8b7dee291488
WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1935
diff
changeset
|
160 log.Printf("info: Paging supported with %d feature per page.\n", |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 featuresPerPage) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 if !op.SupportsHits() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 supportsPaging = false |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 log.Printf("error: %v\n", err) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 supportsPaging = false |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 } else { |
2096
8b7dee291488
WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1935
diff
changeset
|
171 log.Printf("info: Number of features: %d\n", numFeatures) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
176 var downloadURLs []string |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
177 wfs2 := !versionIsLess(wfsVersion, WFS200) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
178 |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
179 addNS := func(v url.Values) { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
180 if len(feature.Namespaces) == 0 { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
181 return |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
182 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
183 // Only use first namespace |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
184 ns := feature.Namespaces[0] |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
185 if wfs2 { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
186 v.Set("NAMESPACES", fmt.Sprintf("(%s,%s)", ns.Space, ns.Local)) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
187 } else { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
188 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local)) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
189 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
190 } |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
191 |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
192 addOutputFormat := func(v url.Values) { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
193 if outputFormat != "" { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
194 v.Set("outputFormat", outputFormat) |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
195 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
196 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
197 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
198 addSortBy := func(v url.Values) { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
199 if sortBy != "" { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
200 v.Set("sortBy", sortBy) |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
201 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
202 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
203 |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
204 if supportsPaging { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
205 pagedURL := func(ofs, count int) string { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
206 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
207 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
208 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
209 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
210 v.Set("startIndex", strconv.Itoa(ofs)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
211 if wfs2 { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 v.Set("count", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
214 v.Set("maxFeatures", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
217 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
218 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
219 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
220 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
221 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
222 return q.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
223 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
224 if numFeatures <= featuresPerPage { |
2096
8b7dee291488
WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1935
diff
changeset
|
225 log.Println("info: All features can be fetched in one page.") |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
226 downloadURLs = []string{pagedURL(0, numFeatures)} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
227 } else { |
2096
8b7dee291488
WFS downloader: Harmonized logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1935
diff
changeset
|
228 log.Println("info: Features need to be downloaded in pages.") |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
229 for pos := 0; pos < numFeatures; { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
230 var count int |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
231 if rest := numFeatures - pos; rest >= numFeatures { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
232 count = numFeatures |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
233 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
234 count = rest |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
235 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
236 downloadURLs = append(downloadURLs, pagedURL(pos, count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
237 pos += count |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
238 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
239 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
240 } else { // No paging support. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
241 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
242 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
243 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
244 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
245 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
246 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
247 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
248 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
250 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
251 downloadURLs = []string{q.String()} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
252 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
254 return downloadURLs, nil |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
256 |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2096
diff
changeset
|
257 func downloadURL(url string, handler func(string, io.Reader) error) error { |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
258 resp, err := http.Get(url) |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
259 if err != nil { |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
260 return err |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
261 } |
1626
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
262 if resp.StatusCode < 200 || resp.StatusCode > 299 { |
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
263 return fmt.Errorf("Invalid HTTP status code: %d (%s)", |
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
264 resp.StatusCode, resp.Status) |
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
265 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
266 defer resp.Body.Close() |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
267 |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
268 var already bytes.Buffer |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
269 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
270 // Prevent the XML reader from consuming everything. |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
271 limit := io.LimitReader(resp.Body, 16*1024) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
272 in := io.TeeReader(limit, &already) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
273 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
274 if err := scanExceptionReport(in); err != nil { |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
275 return err |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
276 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
277 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
278 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
279 return handler(url, multi) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
280 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
281 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
282 type ExceptionReport struct { |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
283 Code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
284 Text string |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
285 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
286 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
287 func (er *ExceptionReport) Error() string { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
288 return fmt.Sprintf( |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
289 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s", |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
290 er.Code, er.Text) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
291 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
292 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
293 func scanExceptionReport(r io.Reader) *ExceptionReport { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
294 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
295 const ows = "http://www.opengis.net/ows/1.1" |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
296 |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
297 decoder := xml.NewDecoder(r) |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
298 decoder.CharsetReader = charset.NewReaderLabel |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
299 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
300 var ( |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
301 isError bool |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
302 code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
303 depth int |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
304 text strings.Builder |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
305 ) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
306 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
307 type tokenFunc func(xml.Token) tokenFunc |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
308 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
309 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
310 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
311 exceptionReportFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
312 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
313 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
314 return exceptionReportFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
315 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
316 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
317 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
318 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
319 isError = true |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
320 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
321 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
322 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
323 exceptionFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
324 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
325 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
326 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
327 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
328 if e.Name.Local == "Exception" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
329 for i := range e.Attr { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
330 at := &e.Attr[i] |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
331 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
332 code = at.Value |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
333 break |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
334 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
335 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
336 return exceptionTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
337 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
338 return exceptionFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
339 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
340 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
341 exceptionTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
342 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
343 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
344 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
345 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
346 return exceptionTextFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
347 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
348 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
349 collectTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
350 switch e := t.(type) { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
351 case xml.StartElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
352 depth++ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
353 case xml.CharData: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
354 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
355 text.Write(e) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
356 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
357 case xml.EndElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
358 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
359 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
360 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
361 depth-- |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
362 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
363 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
364 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
365 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
366 tokens: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
367 for fn := exceptionReportFn; fn != nil; { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
368 tok, err := decoder.Token() |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
369 switch { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
370 case tok == nil && err == io.EOF: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
371 break tokens |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
372 case err != nil: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
373 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
374 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
375 fn = fn(tok) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
376 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
377 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
378 if isError { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
379 return &ExceptionReport{ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
380 Code: code, |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
381 Text: text.String(), |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
382 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
383 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
384 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
385 return nil |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
386 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
387 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
388 // DownloadURLs does the actual GetFeature requests downloads |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
389 // and hands the resulting io.Readers over to the given handler. |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2096
diff
changeset
|
390 func DownloadURLs(urls []string, handler func(string, io.Reader) error) error { |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
391 for _, url := range urls { |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
392 if err := downloadURL(url, handler); err != nil { |
1935
73f9e15d0384
WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1689
diff
changeset
|
393 return err |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
394 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
395 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
396 return nil |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
397 } |