Mercurial > gemma
annotate pkg/wfs/download.go @ 5520:05db984d3db1
Improve performance of bottleneck area calculation
Avoid buffer calculations by replacing them with simple distance comparisons
and calculate the boundary of the result geometry only once per iteration.
In some edge cases with very large numbers of iterations, this reduced
the runtime of a bottleneck import by a factor of more than twenty.
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Thu, 21 Oct 2021 19:50:39 +0200 |
parents | 5f47eeea988d |
children |
rev | line source |
---|---|
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package wfs |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
18 "bytes" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "encoding/xml" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "errors" |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
21 "fmt" |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
22 "io" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "net/http" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "net/url" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "strconv" |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
26 "strings" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 "golang.org/x/net/html/charset" |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
29 |
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/log" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 var ( |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
34 // ErrNoSuchFeatureType is returned when a feature is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
35 ErrNoSuchFeatureType = errors.New("no such feature type") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
36 // ErrGetFeatureNotSupported is returned when GetFeature is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
37 ErrGetFeatureNotSupported = errors.New("method GetFeature not supported") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
38 // ErrMethodGetNotSupported is returned when the GET is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
39 ErrMethodGetNotSupported = errors.New("method GET not supported") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
40 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
41 ErrNoNumberMatchedFound = errors.New("no numberMatched attribute found") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
42 // ErrOutputFormatNotSupported is returned if a output format is |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
43 // not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
44 ErrOutputFormatNotSupported = errors.New("output format not supported") |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
47 // GetCapabilities downloads a capabilities document for a given URL. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 func GetCapabilities(capURL string) (*Capabilities, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 base, err := url.Parse(capURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 v.Set("REQUEST", "GetCapabilities") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 base.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 baseURL := base.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 resp, err := http.Get(baseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 caps, err := ParseCapabilities(bufio.NewReader(resp.Body)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 if err == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 caps.BaseURL = baseURL |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 return caps, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 v.Set("resultType", "hits") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 v.Set("VERSION", version) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 v.Set("TYPENAMES", featureType) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 q := *u |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 resp, err := http.Get(q.String()) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 dec := xml.NewDecoder(resp.Body) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 dec.CharsetReader = charset.NewReaderLabel |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 var result struct { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 NumberMatched *int `xml:"numberMatched,attr"` |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 if err := dec.Decode(&result); err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 if result.NumberMatched == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 return 0, ErrNoNumberMatchedFound |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 return *result.NumberMatched, nil |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
108 // GetFeaturesGET constructs a list of URLs to get features |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
109 // for a given feature type from a WFS servers. |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
110 func GetFeaturesGET( |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
111 caps *Capabilities, |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
112 featureTypeName string, |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
113 outputFormats []string, |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
114 sortBy string, |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
115 ) ([]string, error) { |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
116 |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
117 feature := caps.FindFeatureType(featureTypeName) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
118 if feature == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
119 return nil, ErrNoSuchFeatureType |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
120 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
121 op := caps.FindOperation("GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
122 if op == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
123 return nil, ErrGetFeatureNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
124 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
125 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
126 if op.DCP.HTTP.Get == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
127 return nil, ErrMethodGetNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
128 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
129 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
130 getRaw := op.DCP.HTTP.Get.HRef |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
131 getU, err := url.Parse(getRaw) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
132 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
133 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
134 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
135 // The URL could be relative so resolve against Capabilities URL. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
136 if !getU.IsAbs() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
137 base, err := url.Parse(caps.BaseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
138 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
139 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
140 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
141 getU = getU.ResolveReference(base) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
142 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
144 var outputFormat string |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
145 |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
146 if len(outputFormats) > 0 { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
147 if outputFormat = op.SupportsOutputFormat(outputFormats...); outputFormat == "" { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
148 return nil, ErrOutputFormatNotSupported |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
149 } |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
150 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
151 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
152 wfsVersion := caps.HighestWFSVersion(WFS200) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 featuresPerPage, supportsPaging := op.FeaturesPerPage() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 var numFeatures int |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 if supportsPaging { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
159 log.Infof("Paging supported with %d feature per page.\n", |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 featuresPerPage) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 if !op.SupportsHits() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 supportsPaging = false |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 if err != nil { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
167 log.Errorf("%v\n", err) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 supportsPaging = false |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
169 } else if numFeatures == 0 { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
170 return nil, nil |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 } else { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
172 log.Infof("Number of features: %d\n", numFeatures) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
176 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
177 var downloadURLs []string |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
178 wfs2 := !versionIsLess(wfsVersion, WFS200) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
179 |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
180 addNS := func(v url.Values) { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
181 if len(feature.Namespaces) == 0 { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
182 return |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
183 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
184 // Only use first namespace |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
185 ns := feature.Namespaces[0] |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
186 if wfs2 { |
4880
ed0de319d09a
Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents:
4166
diff
changeset
|
187 v.Set( |
ed0de319d09a
Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents:
4166
diff
changeset
|
188 "NAMESPACES", fmt.Sprintf("xmlns(%s,%s)", ns.Space, ns.Local)) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
189 } else { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
190 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local)) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
191 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
192 } |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
194 addOutputFormat := func(v url.Values) { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
195 if outputFormat != "" { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
196 v.Set("outputFormat", outputFormat) |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
197 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
198 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
199 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
200 addSortBy := func(v url.Values) { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
201 if sortBy != "" { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
202 v.Set("sortBy", sortBy) |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
203 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
204 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
205 |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
206 if supportsPaging { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
207 pagedURL := func(ofs, count int) string { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
208 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
209 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
210 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
211 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 v.Set("startIndex", strconv.Itoa(ofs)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 if wfs2 { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
214 v.Set("count", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 v.Set("maxFeatures", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
218 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
219 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
220 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
221 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
222 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
223 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
224 return q.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
225 } |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
226 if numFeatures > 0 { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
227 if numFeatures <= featuresPerPage { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
228 log.Infof("all features can be fetched in one page.") |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
229 downloadURLs = []string{pagedURL(0, numFeatures)} |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
230 } else { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
231 log.Infof("features need to be downloaded in pages.") |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
232 for pos := 0; pos < numFeatures; { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
233 var count int |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
234 if rest := numFeatures - pos; rest >= numFeatures { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
235 count = numFeatures |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
236 } else { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
237 count = rest |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
238 } |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
239 downloadURLs = append(downloadURLs, pagedURL(pos, count)) |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
240 pos += count |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
241 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
242 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
243 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
244 } else { // No paging support. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
245 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
246 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
247 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
248 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
250 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
251 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
252 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
254 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 downloadURLs = []string{q.String()} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
257 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
258 return downloadURLs, nil |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
259 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
260 |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
261 func downloadURL(user, password, url string, handler func(string, io.Reader) error) error { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
262 req, err := http.NewRequest(http.MethodGet, url, nil) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
263 if err != nil { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
264 return err |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
265 } |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
266 |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
267 if user != "" || password != "" { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
268 req.SetBasicAuth(user, password) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
269 } |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
270 |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
271 resp, err := http.DefaultClient.Do(req) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
272 |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
273 if err != nil { |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
274 return err |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
275 } |
1626
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
276 if resp.StatusCode < 200 || resp.StatusCode > 299 { |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
277 return fmt.Errorf("invalid HTTP status code: %d (%s)", |
1626
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
278 resp.StatusCode, resp.Status) |
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
279 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
280 defer resp.Body.Close() |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
281 |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
282 var already bytes.Buffer |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
283 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
284 // Prevent the XML reader from consuming everything. |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
285 limit := io.LimitReader(resp.Body, 16*1024) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
286 in := io.TeeReader(limit, &already) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
287 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
288 if err := scanExceptionReport(in); err != nil { |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
289 return err |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
290 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
291 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
292 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
293 return handler(url, multi) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
294 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
295 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
296 // ExceptionReport is an error with the extract code and |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
297 // text from an OWS exception document. |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
298 type ExceptionReport struct { |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
299 Code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
300 Text string |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
301 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
302 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
303 func (er *ExceptionReport) Error() string { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
304 return fmt.Sprintf( |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
305 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s", |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
306 er.Code, er.Text) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
307 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
308 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
309 func scanExceptionReport(r io.Reader) *ExceptionReport { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
310 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
311 const ows = "http://www.opengis.net/ows/1.1" |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
312 |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
313 decoder := xml.NewDecoder(r) |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
314 decoder.CharsetReader = charset.NewReaderLabel |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
315 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
316 var ( |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
317 isError bool |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
318 code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
319 depth int |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
320 text strings.Builder |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
321 ) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
322 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
323 type tokenFunc func(xml.Token) tokenFunc |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
324 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
325 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
326 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
327 exceptionReportFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
328 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
329 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
330 return exceptionReportFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
331 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
332 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
333 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
334 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
335 isError = true |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
336 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
337 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
338 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
339 exceptionFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
340 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
341 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
342 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
343 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
344 if e.Name.Local == "Exception" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
345 for i := range e.Attr { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
346 at := &e.Attr[i] |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
347 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
348 code = at.Value |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
349 break |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
350 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
351 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
352 return exceptionTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
353 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
354 return exceptionFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
355 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
356 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
357 exceptionTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
358 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
359 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
360 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
361 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
362 return exceptionTextFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
363 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
364 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
365 collectTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
366 switch e := t.(type) { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
367 case xml.StartElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
368 depth++ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
369 case xml.CharData: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
370 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
371 text.Write(e) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
372 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
373 case xml.EndElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
374 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
375 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
376 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
377 depth-- |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
378 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
379 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
380 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
381 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
382 tokens: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
383 for fn := exceptionReportFn; fn != nil; { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
384 tok, err := decoder.Token() |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
385 switch { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
386 case tok == nil && err == io.EOF: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
387 break tokens |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
388 case err != nil: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
389 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
390 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
391 fn = fn(tok) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
392 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
393 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
394 if isError { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
395 return &ExceptionReport{ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
396 Code: code, |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
397 Text: text.String(), |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
398 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
399 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
400 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
401 return nil |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
402 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
403 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
404 // DownloadURLs does the actual GetFeature requests downloads |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
405 // and hands the resulting io.Readers over to the given handler. |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
406 func DownloadURLs(user, password string, urls []string, handler func(string, io.Reader) error) error { |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
407 for _, url := range urls { |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
408 if err := downloadURL(user, password, url, handler); err != nil { |
1935
73f9e15d0384
WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1689
diff
changeset
|
409 return err |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
410 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
411 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
412 return nil |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
413 } |