Mercurial > gemma
annotate pkg/wfs/download.go @ 5641:c0425c948c14 DQR adjustments
fix copy&paste error
author | Thomas Junk <thomas.junk@intevation.de> |
---|---|
date | Thu, 01 Jun 2023 12:36:50 +0200 |
parents | 5f47eeea988d |
children |
rev | line source |
---|---|
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0.Reader. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package wfs |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
18 "bytes" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "encoding/xml" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "errors" |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
21 "fmt" |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
22 "io" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "net/http" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "net/url" |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "strconv" |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
26 "strings" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 "golang.org/x/net/html/charset" |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
29 |
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/log" |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 var ( |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
34 // ErrNoSuchFeatureType is returned when a feature is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
35 ErrNoSuchFeatureType = errors.New("no such feature type") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
36 // ErrGetFeatureNotSupported is returned when GetFeature is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
37 ErrGetFeatureNotSupported = errors.New("method GetFeature not supported") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
38 // ErrMethodGetNotSupported is returned when the GET is not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
39 ErrMethodGetNotSupported = errors.New("method GET not supported") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
40 // ErrNoNumberMatchedFound is returned if feature count cannot be extracted. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
41 ErrNoNumberMatchedFound = errors.New("no numberMatched attribute found") |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
42 // ErrOutputFormatNotSupported is returned if a output format is |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
43 // not supported. |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
44 ErrOutputFormatNotSupported = errors.New("output format not supported") |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 ) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
47 // GetCapabilities downloads a capabilities document for a given URL. |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 func GetCapabilities(capURL string) (*Capabilities, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 base, err := url.Parse(capURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 v.Set("REQUEST", "GetCapabilities") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 v.Set("ACCEPTVERSIONS", "2.0.0,1.1.0,1.0.0") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 base.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 baseURL := base.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 resp, err := http.Get(baseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 return nil, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 caps, err := ParseCapabilities(bufio.NewReader(resp.Body)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 if err == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 caps.BaseURL = baseURL |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 return caps, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 func numberFeaturesGET(u *url.URL, featureType, version string) (int, error) { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 v.Set("resultType", "hits") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 v.Set("VERSION", version) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 v.Set("TYPENAMES", featureType) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 q := *u |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 resp, err := http.Get(q.String()) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 if err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 defer resp.Body.Close() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 dec := xml.NewDecoder(resp.Body) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 dec.CharsetReader = charset.NewReaderLabel |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 var result struct { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 NumberMatched *int `xml:"numberMatched,attr"` |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 if err := dec.Decode(&result); err != nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 return 0, err |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 if result.NumberMatched == nil { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 return 0, ErrNoNumberMatchedFound |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 return *result.NumberMatched, nil |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
108 // GetFeaturesGET constructs a list of URLs to get features |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
109 // for a given feature type from a WFS servers. |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
110 func GetFeaturesGET( |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
111 caps *Capabilities, |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
112 featureTypeName string, |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
113 outputFormats []string, |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
114 sortBy string, |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
115 ) ([]string, error) { |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
116 |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
117 feature := caps.FindFeatureType(featureTypeName) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
118 if feature == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
119 return nil, ErrNoSuchFeatureType |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
120 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
121 op := caps.FindOperation("GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
122 if op == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
123 return nil, ErrGetFeatureNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
124 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
125 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
126 if op.DCP.HTTP.Get == nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
127 return nil, ErrMethodGetNotSupported |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
128 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
129 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
130 getRaw := op.DCP.HTTP.Get.HRef |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
131 getU, err := url.Parse(getRaw) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
132 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
133 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
134 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
135 // The URL could be relative so resolve against Capabilities URL. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
136 if !getU.IsAbs() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
137 base, err := url.Parse(caps.BaseURL) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
138 if err != nil { |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
139 return nil, err |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
140 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
141 getU = getU.ResolveReference(base) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
142 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 |
2622
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
144 var outputFormat string |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
145 |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
146 if len(outputFormats) > 0 { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
147 if outputFormat = op.SupportsOutputFormat(outputFormats...); outputFormat == "" { |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
148 return nil, ErrOutputFormatNotSupported |
02505fcff63c
WFS downloads: Accept 'geojson' as mime type, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2620
diff
changeset
|
149 } |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
150 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
151 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
152 wfsVersion := caps.HighestWFSVersion(WFS200) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
154 featuresPerPage, supportsPaging := op.FeaturesPerPage() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 var numFeatures int |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 if supportsPaging { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
159 log.Infof("Paging supported with %d feature per page.\n", |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 featuresPerPage) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 if !op.SupportsHits() { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 supportsPaging = false |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 numFeatures, err = numberFeaturesGET(getU, featureTypeName, wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 if err != nil { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
167 log.Errorf("%v\n", err) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 supportsPaging = false |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
169 } else if numFeatures == 0 { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
170 return nil, nil |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 } else { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
172 log.Infof("Number of features: %d\n", numFeatures) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
176 |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
177 var downloadURLs []string |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
178 wfs2 := !versionIsLess(wfsVersion, WFS200) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
179 |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
180 addNS := func(v url.Values) { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
181 if len(feature.Namespaces) == 0 { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
182 return |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
183 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
184 // Only use first namespace |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
185 ns := feature.Namespaces[0] |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
186 if wfs2 { |
4880
ed0de319d09a
Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents:
4166
diff
changeset
|
187 v.Set( |
ed0de319d09a
Fix parameter format for WFS 2.0
Tom Gottfried <tom@intevation.de>
parents:
4166
diff
changeset
|
188 "NAMESPACES", fmt.Sprintf("xmlns(%s,%s)", ns.Space, ns.Local)) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
189 } else { |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
190 v.Set("NAMESPACE", fmt.Sprintf("(%s:%s)", ns.Space, ns.Local)) |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
191 } |
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
192 } |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
194 addOutputFormat := func(v url.Values) { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
195 if outputFormat != "" { |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
196 v.Set("outputFormat", outputFormat) |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
197 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
198 } |
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
199 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
200 addSortBy := func(v url.Values) { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
201 if sortBy != "" { |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
202 v.Set("sortBy", sortBy) |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
203 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
204 } |
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
205 |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
206 if supportsPaging { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
207 pagedURL := func(ofs, count int) string { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
208 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
209 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
210 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
211 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 v.Set("startIndex", strconv.Itoa(ofs)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 if wfs2 { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
214 v.Set("count", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 } else { |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 v.Set("maxFeatures", strconv.Itoa(count)) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
218 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
219 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
220 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
221 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
222 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
223 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
224 return q.String() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
225 } |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
226 if numFeatures > 0 { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
227 if numFeatures <= featuresPerPage { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
228 log.Infof("all features can be fetched in one page.") |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
229 downloadURLs = []string{pagedURL(0, numFeatures)} |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
230 } else { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
231 log.Infof("features need to be downloaded in pages.") |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
232 for pos := 0; pos < numFeatures; { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
233 var count int |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
234 if rest := numFeatures - pos; rest >= numFeatures { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
235 count = numFeatures |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
236 } else { |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
237 count = rest |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
238 } |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
239 downloadURLs = append(downloadURLs, pagedURL(pos, count)) |
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4880
diff
changeset
|
240 pos += count |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
241 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
242 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
243 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
244 } else { // No paging support. |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
245 v := url.Values{} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
246 v.Set("SERVICE", "WFS") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
247 v.Set("REQUEST", "GetFeature") |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
248 v.Set("VERSION", wfsVersion) |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 v.Set("TYPENAMES", featureTypeName) |
1614
efc409e330a6
WFS downloader: Propagate namespaces of feature type into GetFeature request.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1608
diff
changeset
|
250 addNS(v) |
1616
f59550310143
WFS downloader: Support different feature output formats.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1614
diff
changeset
|
251 addOutputFormat(v) |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
252 addSortBy(v) |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 q := *getU |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
254 q.RawQuery = v.Encode() |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 downloadURLs = []string{q.String()} |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 } |
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
257 |
1619
3093bab05c81
WFS downloader: Dump features to stdout for testing purposes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1616
diff
changeset
|
258 return downloadURLs, nil |
1608
427f9010b4a9
WFS download: Started with GET downloader (paged and unpaged).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
259 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
260 |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
261 func downloadURL(user, password, url string, handler func(string, io.Reader) error) error { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
262 req, err := http.NewRequest(http.MethodGet, url, nil) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
263 if err != nil { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
264 return err |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
265 } |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
266 |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
267 if user != "" || password != "" { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
268 req.SetBasicAuth(user, password) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
269 } |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
270 |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
271 resp, err := http.DefaultClient.Do(req) |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
272 |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
273 if err != nil { |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
274 return err |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
275 } |
1626
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
276 if resp.StatusCode < 200 || resp.StatusCode > 299 { |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
277 return fmt.Errorf("invalid HTTP status code: %d (%s)", |
1626
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
278 resp.StatusCode, resp.Status) |
92da44ba610c
WFS downloader: Parse into raw feature collections.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1624
diff
changeset
|
279 } |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
280 defer resp.Body.Close() |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
281 |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
282 var already bytes.Buffer |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
283 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
284 // Prevent the XML reader from consuming everything. |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
285 limit := io.LimitReader(resp.Body, 16*1024) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
286 in := io.TeeReader(limit, &already) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
287 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
288 if err := scanExceptionReport(in); err != nil { |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
289 return err |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
290 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
291 |
2110
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
292 multi := io.MultiReader(bytes.NewReader(already.Bytes()), resp.Body) |
fc4ebe7eaeff
WFS downloader: Don't use temp files. TODO: Use a streaming scanner for the detection of the exception documents.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2109
diff
changeset
|
293 return handler(url, multi) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
294 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
295 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
296 // ExceptionReport is an error with the extract code and |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
297 // text from an OWS exception document. |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
298 type ExceptionReport struct { |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
299 Code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
300 Text string |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
301 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
302 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
303 func (er *ExceptionReport) Error() string { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
304 return fmt.Sprintf( |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
305 "WFS GetFeature error: ExceptionCode: '%s' / ExceptionText: %s", |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
306 er.Code, er.Text) |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
307 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
308 |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
309 func scanExceptionReport(r io.Reader) *ExceptionReport { |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
310 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
311 const ows = "http://www.opengis.net/ows/1.1" |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
312 |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
313 decoder := xml.NewDecoder(r) |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
314 decoder.CharsetReader = charset.NewReaderLabel |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
315 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
316 var ( |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
317 isError bool |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
318 code string |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
319 depth int |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
320 text strings.Builder |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
321 ) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
322 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
323 type tokenFunc func(xml.Token) tokenFunc |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
324 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
325 var exceptionReportFn, exceptionFn, exceptionTextFn, collectTextFn tokenFunc |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
326 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
327 exceptionReportFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
328 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
329 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
330 return exceptionReportFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
331 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
332 if e.Name.Local != "ExceptionReport" && e.Name.Space != ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
333 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
334 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
335 isError = true |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
336 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
337 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
338 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
339 exceptionFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
340 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
341 if !ok { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
342 return exceptionFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
343 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
344 if e.Name.Local == "Exception" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
345 for i := range e.Attr { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
346 at := &e.Attr[i] |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
347 if at.Name.Local == "code" || at.Name.Local == "exceptionCode" { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
348 code = at.Value |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
349 break |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
350 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
351 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
352 return exceptionTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
353 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
354 return exceptionFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
355 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
356 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
357 exceptionTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
358 e, ok := t.(xml.StartElement) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
359 if ok && e.Name.Local == "ExceptionText" && e.Name.Space == ows { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
360 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
361 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
362 return exceptionTextFn |
2109
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
363 } |
86c88fc0ff5e
WFS downloader: Parse for exception reports when calling GetFeature. Not very efficient (uses temp files).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2106
diff
changeset
|
364 |
2118
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
365 collectTextFn = func(t xml.Token) tokenFunc { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
366 switch e := t.(type) { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
367 case xml.StartElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
368 depth++ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
369 case xml.CharData: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
370 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
371 text.Write(e) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
372 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
373 case xml.EndElement: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
374 if depth == 0 { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
375 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
376 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
377 depth-- |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
378 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
379 return collectTextFn |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
380 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
381 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
382 tokens: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
383 for fn := exceptionReportFn; fn != nil; { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
384 tok, err := decoder.Token() |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
385 switch { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
386 case tok == nil && err == io.EOF: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
387 break tokens |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
388 case err != nil: |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
389 return nil |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
390 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
391 fn = fn(tok) |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
392 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
393 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
394 if isError { |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
395 return &ExceptionReport{ |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
396 Code: code, |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
397 Text: text.String(), |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
398 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
399 } |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
400 |
8a6ed058af22
WFS downloader: Use a custom XML streaming parser to detect error reports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2110
diff
changeset
|
401 return nil |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
402 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
403 |
1689
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
404 // DownloadURLs does the actual GetFeature requests downloads |
6caf5cd6249e
WFS: Made golint happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1626
diff
changeset
|
405 // and hands the resulting io.Readers over to the given handler. |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
406 func DownloadURLs(user, password string, urls []string, handler func(string, io.Reader) error) error { |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
407 for _, url := range urls { |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
408 if err := downloadURL(user, password, url, handler); err != nil { |
1935
73f9e15d0384
WFS downloader: die correctly if an error was detected when downloading features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1689
diff
changeset
|
409 return err |
1624
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
410 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
411 } |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
412 return nil |
943823d03d50
WFS downloader: Started with mapping return features to Go structs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1619
diff
changeset
|
413 } |