Mercurial > gemma
annotate pkg/misc/http.go @ 5717:1c3bdd114ca1 errors-in-csv-export
Add errors column to CSV export.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Mon, 11 Dec 2023 16:16:57 +0100 |
parents | 1222b777f51f |
children | 31973f6f5cca |
rev | line source |
---|---|
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package misc |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "bufio" |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
18 "fmt" |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "io" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "io/ioutil" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "net/http" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "os" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "path/filepath" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "gemma.intevation.de/gemma/pkg/config" |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 ) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 |
4172
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
28 // StoreUploadedFile stores a file upload file from |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
29 // a given HTTP request identified by a given field name |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
30 // in a file with a path build by the config.TmpDir and |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
31 // the given file name. |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
32 // If the file is long than the given limit maxSize |
9cbed444b8a4
Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2194
diff
changeset
|
33 // this function returns an error. |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
34 func StoreUploadedFile( |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
35 req *http.Request, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
36 field, fname string, |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
37 maxSize int64) (string, error) { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
38 return StoreUploadedFileCheck(req, field, fname, maxSize, false) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
39 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
40 |
5601
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
41 // StoreUploadedFileCheck does the same as StoreUploadedFile |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
42 // with the optional check if the upload is too large or not. |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
43 func StoreUploadedFileCheck( |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
44 req *http.Request, |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
45 field, fname string, |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
46 maxSize int64, |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
47 errorOverMax bool, |
1222b777f51f
Made golint finally happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4875
diff
changeset
|
48 ) (string, error) { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 // Check for direct upload. |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 f, _, err := req.FormFile(field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 defer f.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 dir, err := ioutil.TempDir(config.TmpDir(), field) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 o, err := os.Create(filepath.Join(dir, fname)) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 if err != nil { |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 out := bufio.NewWriter(o) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
70 // Little trick to check if we are over the limit. |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
71 size := maxSize |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
72 if errorOverMax { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
73 size++ |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
74 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
75 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
76 cleanup := func() { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 o.Close() |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 os.RemoveAll(dir) |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
79 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
80 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
81 r, err := io.Copy(out, io.LimitReader(f, size)) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
82 if err != nil { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
83 cleanup() |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
87 if errorOverMax && r > maxSize { |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
88 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
89 return "", fmt.Errorf("upload exceeded limit of %d bytes", maxSize) |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
90 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
91 |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 if err = out.Flush(); err != nil { |
4875
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
93 cleanup() |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
94 return "", err |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
95 } |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
96 |
6237e6165041
* Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4172
diff
changeset
|
97 if err := o.Close(); err != nil { |
2194
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 os.RemoveAll(dir) |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 return "", err |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 } |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 return dir, nil |
4d6979dedb11
Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 } |