Mercurial > gemma
annotate pkg/wfs/global.go @ 5520:05db984d3db1
Improve performance of bottleneck area calculation
Avoid buffer calculations by replacing them with simple distance comparisons
and calculate the boundary of the result geometry only once per iteration.
In some edge cases with very large numbers of iterations, this reduced
the runtime of a bottleneck import by a factor of more than twenty.
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Thu, 21 Oct 2021 19:50:39 +0200 |
parents | 09346efa7f69 |
children | 31973f6f5cca |
rev | line source |
---|---|
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2019 by via donau |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package wfs |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
17 "bufio" |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
18 "fmt" |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "io" |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
20 "io/ioutil" |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
21 "os" |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "os/exec" |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
23 "path/filepath" |
5491
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
24 "sync" |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
25 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
26 "gemma.intevation.de/gemma/pkg/config" |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/log" |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 ) |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
30 var ( |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
31 // FormatGeoJSON is a list of MIME types of |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
32 // GeoJSON documents. |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
33 FormatGeoJSON = []string{"geojson", "application/json"} |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
34 // FormatGML is a list of MIME types |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
35 // GML documents. |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
36 FormatGML = []string{ |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
37 "application/gml+xml; version=3.2", |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
38 "gml2", "gml3", "gml32", |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
39 "text/xml; subtype=gml/2.1.2", |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
40 "text/xml; subtype=gml/3.1.1", |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
41 "text/xml; subtype=gml/3.2", |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
42 } |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
43 ) |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
44 |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 type ( |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
46 // Downloader abstracts the idea of a download function for WFS. |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 Downloader interface { |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2711
diff
changeset
|
48 Download(user, password string, handler func(string, io.Reader) error) error |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 } |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
51 // GeoJSONDownloader is a Downloader for GeoJSON. |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 GeoJSONDownloader []string |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
53 // GMLDownloader is a Downloader for GML. |
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
54 GMLDownloader []string |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 ) |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 |
5491
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
57 var ( |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
58 getFeaturesOnce sync.Once |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
59 getFeatures func(*Capabilities, string, string) (Downloader, error) |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
60 ) |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
5491
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
62 func getFeaturesOnceFunc() { |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 path, err := exec.LookPath("ogr2ogr") |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 if err != nil { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4990
diff
changeset
|
65 log.Infoln("ogr2ogr not installed. Using direct GeoJSON WFS download.") |
5491
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
66 getFeatures = getFeaturesGeoJSON |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
67 } else { |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
68 log.Infof("ogr2ogr found at %s. Using GML WFS download.\n", path) |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
69 getFeatures = getFeaturesGML |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 } |
5491
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
71 } |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
72 |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
73 // GetFeatures is the default Downloader in this Gemma server. |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
74 func GetFeatures( |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
75 caps *Capabilities, |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
76 featureTypeName string, |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
77 sortBy string, |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
78 ) (Downloader, error) { |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
79 getFeaturesOnce.Do(getFeaturesOnceFunc) |
09346efa7f69
Do ogr2ogr check only once when it's needed the first time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5490
diff
changeset
|
80 return getFeatures(caps, featureTypeName, sortBy) |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 } |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
83 func getFeaturesGeoJSON( |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 caps *Capabilities, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 featureTypeName string, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 sortBy string, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 ) (Downloader, error) { |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
88 urls, err := GetFeaturesGET( |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
89 caps, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
90 featureTypeName, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
91 FormatGeoJSON, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
92 sortBy) |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
93 return GeoJSONDownloader(urls), err |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 } |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
96 func getFeaturesGML( |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 caps *Capabilities, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 featureTypeName string, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 sortBy string, |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 ) (Downloader, error) { |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
101 urls, err := GetFeaturesGET( |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
102 caps, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
103 featureTypeName, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
104 FormatGML, |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
105 sortBy) |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
106 return GMLDownloader(urls), err |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 } |
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
108 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
109 // Download is the GeoJSON implementation of the Downloader. |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2711
diff
changeset
|
110 func (gjd GeoJSONDownloader) Download(user, password string, handler func(string, io.Reader) error) error { |
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2711
diff
changeset
|
111 return DownloadURLs(user, password, []string(gjd), handler) |
2702
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
112 } |
d7ef169fd0d3
WFS downloader: More on downloading GML. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2634
diff
changeset
|
113 |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
114 func places(n int) int { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
115 places := 1 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
116 if n < 0 { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
117 places++ |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
118 n = -n |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
119 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
120 for value := 10; n >= value; value *= 10 { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
121 places++ |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
122 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
123 return places |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
124 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
125 |
4166
04876d865528
Made 'golint' and 'staticcheck' happy with wfs package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
126 // Download is GML implementaion of the Downloader. |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2711
diff
changeset
|
127 func (gmd GMLDownloader) Download(user, password string, handler func(string, io.Reader) error) error { |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
128 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
129 if len(gmd) == 0 { |
4990
16259efa828f
Dont try to download from WFS if they dont have features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4166
diff
changeset
|
130 return nil |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
131 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
132 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
133 tmpDir := config.TmpDir() |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
134 dlDir, err := ioutil.TempDir(tmpDir, "wfs-downloads") |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
135 if err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
136 return err |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
137 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
138 defer os.RemoveAll(dlDir) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
139 digits := places(len(gmd)) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
140 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
141 var files []string |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
142 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
143 for i, url := range gmd { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
144 fname := filepath.Join(dlDir, fmt.Sprintf("%0*d.gml", digits, i)) |
2719
2b6f73c30016
WFS downloader: Add BasicAuth support for downloading. TODO: Use it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2711
diff
changeset
|
145 if err := downloadURL(user, password, url, func(_ string, r io.Reader) error { |
2711
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
146 f, err := os.Create(fname) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
147 if err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
148 return err |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
149 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
150 if _, err = io.Copy(f, r); err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
151 f.Close() |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
152 return err |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
153 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
154 return f.Close() |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
155 }); err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
156 return err |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
157 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
158 files = append(files, fname) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
159 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
160 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
161 out := filepath.Join(dlDir, "out.geojson") |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
162 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
163 cmd := exec.Command("ogr2ogr", append([]string{"-f", "GeoJSON", out}, files...)...) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
164 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
165 msg, err := cmd.CombinedOutput() |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
166 if err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
167 return fmt.Errorf("ogr2ogr failed: %s", msg) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
168 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
169 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
170 in, err := os.Open(out) |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
171 if err != nil { |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
172 return err |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
173 } |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
174 defer in.Close() |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
175 |
3956de9b6b32
WFS downloader: Implemented GML download to file, ogr2ogr to GeoJSON.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2702
diff
changeset
|
176 return handler(gmd[0], bufio.NewReader(in)) |
2634
49ce3c11ca72
WFS downloads: Started with GML based downloads based on installation of ogr2ogr.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
177 } |