Mercurial > gemma
annotate pkg/misc/http.go @ 5101:1b0b13e70bc1 queued-stage-done
Proxy the original job creator directly and not only the dependencies.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Tue, 24 Mar 2020 16:47:49 +0100 |
parents | 6237e6165041 |
children | 1222b777f51f |
rev | line source |
---|---|
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package misc |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
18 "fmt" |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "io" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "io/ioutil" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "net/http" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "os" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "path/filepath" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "gemma.intevation.de/gemma/pkg/config" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 ) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 |
4172
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
28 // StoreUploadedFile stores a file upload file from |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
29 // a given HTTP request identified by a given field name |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
30 // in a file with a path build by the config.TmpDir and |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
31 // the given file name. |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
32 // If the file is long than the given limit maxSize |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
33 // this function returns an error. |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
34 func StoreUploadedFile( |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
35 req *http.Request, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
36 field, fname string, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
37 maxSize int64) (string, error) { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
38 return StoreUploadedFileCheck(req, field, fname, maxSize, false) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
39 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
40 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
41 func StoreUploadedFileCheck(req *http.Request, field, fname string, maxSize int64, errorOverMax bool) (string, error) { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 // Check for direct upload. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 f, _, err := req.FormFile(field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 defer f.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 dir, err := ioutil.TempDir(config.TmpDir(), field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 o, err := os.Create(filepath.Join(dir, fname)) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 out := bufio.NewWriter(o) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
63 // Little trick to check if we are over the limit. |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
64 size := maxSize |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
65 if errorOverMax { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
66 size++ |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
67 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
68 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
69 cleanup := func() { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 o.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 os.RemoveAll(dir) |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
72 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
73 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
74 r, err := io.Copy(out, io.LimitReader(f, size)) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
75 if err != nil { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
76 cleanup() |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
80 if errorOverMax && r > maxSize { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
81 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
82 return "", fmt.Errorf("upload exceeded limit of %d bytes", maxSize) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
83 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
84 |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 if err = out.Flush(); err != nil { |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
86 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
87 return "", err |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
88 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
89 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
90 if err := o.Close(); err != nil { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 return dir, nil |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 } |