Mercurial > gemma
annotate pkg/misc/http.go @ 5591:0011f50cf216 surveysperbottleneckid
Removed no longer used alternative api for surveys/ endpoint.
As bottlenecks in the summary for SR imports are now identified by
their id and no longer by the (not guarantied to be unique!) name,
there is no longer the need to request survey data by the name+date
tuple (which isn't reliable anyway). So the workaround was now
reversed.
author | Sascha Wilde <wilde@sha-bang.de> |
---|---|
date | Wed, 06 Apr 2022 13:30:29 +0200 |
parents | 6237e6165041 |
children | 1222b777f51f |
rev | line source |
---|---|
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package misc |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
18 "fmt" |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "io" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "io/ioutil" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "net/http" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "os" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "path/filepath" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "gemma.intevation.de/gemma/pkg/config" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 ) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 |
4172
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
28 // StoreUploadedFile stores a file upload file from |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
29 // a given HTTP request identified by a given field name |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
30 // in a file with a path build by the config.TmpDir and |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
31 // the given file name. |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
32 // If the file is long than the given limit maxSize |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
33 // this function returns an error. |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
34 func StoreUploadedFile( |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
35 req *http.Request, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
36 field, fname string, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
37 maxSize int64) (string, error) { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
38 return StoreUploadedFileCheck(req, field, fname, maxSize, false) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
39 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
40 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
41 func StoreUploadedFileCheck(req *http.Request, field, fname string, maxSize int64, errorOverMax bool) (string, error) { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 // Check for direct upload. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 f, _, err := req.FormFile(field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 defer f.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 dir, err := ioutil.TempDir(config.TmpDir(), field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 o, err := os.Create(filepath.Join(dir, fname)) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 out := bufio.NewWriter(o) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
63 // Little trick to check if we are over the limit. |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
64 size := maxSize |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
65 if errorOverMax { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
66 size++ |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
67 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
68 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
69 cleanup := func() { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 o.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 os.RemoveAll(dir) |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
72 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
73 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
74 r, err := io.Copy(out, io.LimitReader(f, size)) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
75 if err != nil { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
76 cleanup() |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
80 if errorOverMax && r > maxSize { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
81 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
82 return "", fmt.Errorf("upload exceeded limit of %d bytes", maxSize) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
83 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
84 |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 if err = out.Flush(); err != nil { |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
86 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
87 return "", err |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
88 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
89 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
90 if err := o.Close(); err != nil { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 return dir, nil |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 } |