Mercurial > gemma

annotate pkg/misc/http.go @ 5591:0011f50cf216 surveysperbottleneckid

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.
author Sascha Wilde <wilde@sha-bang.de>
date Wed, 06 Apr 2022 13:30:29 +0200
parents 6237e6165041
children 1222b777f51f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package misc
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "bufio"
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
18 "fmt"
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "io"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "io/ioutil"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "net/http"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "os"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "path/filepath"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/config"
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 )
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
27
4172
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
28 // StoreUploadedFile stores a file upload file from
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
29 // a given HTTP request identified by a given field name
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
30 // in a file with a path build by the config.TmpDir and
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
31 // the given file name.
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
32 // If the file is long than the given limit maxSize
9cbed444b8a4 Made 'golint' and 'staticcheck' happy with misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2194
diff changeset
33 // this function returns an error.
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
34 func StoreUploadedFile(
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
35 req *http.Request,
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
36 field, fname string,
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
37 maxSize int64) (string, error) {
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
38 return StoreUploadedFileCheck(req, field, fname, maxSize, false)
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
39 }
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
40
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
41 func StoreUploadedFileCheck(req *http.Request, field, fname string, maxSize int64, errorOverMax bool) (string, error) {
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 // Check for direct upload.
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 f, _, err := req.FormFile(field)
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 if err != nil {
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 return "", err
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 }
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48 defer f.Close()
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 dir, err := ioutil.TempDir(config.TmpDir(), field)
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 if err != nil {
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 return "", err
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 }
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 o, err := os.Create(filepath.Join(dir, fname))
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 if err != nil {
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 os.RemoveAll(dir)
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 return "", err
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 }
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 out := bufio.NewWriter(o)
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
63 // Little trick to check if we are over the limit.
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
64 size := maxSize
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
65 if errorOverMax {
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
66 size++
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
67 }
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
68
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
69 cleanup := func() {
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 o.Close()
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71 os.RemoveAll(dir)
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
72 }
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
73
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
74 r, err := io.Copy(out, io.LimitReader(f, size))
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
75 if err != nil {
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
76 cleanup()
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77 return "", err
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 }
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
80 if errorOverMax && r > maxSize {
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
81 cleanup()
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
82 return "", fmt.Errorf("upload exceeded limit of %d bytes", maxSize)
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
83 }
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
84
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85 if err = out.Flush(); err != nil {
4875
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
86 cleanup()
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
87 return "", err
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
88 }
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
89
6237e6165041 * Raise the upload limit for sounding results from 25GB up to 50GB.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4172
diff changeset
90 if err := o.Close(); err != nil {
2194
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
91 os.RemoveAll(dir)
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92 return "", err
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93 }
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95 return dir, nil
4d6979dedb11 Imports: Deduplicted file upload code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96 }