gemma: pkg/imports/statsupdate.go annotate

annotate pkg/imports/statsupdate.go @ 5718:3d497077f888 uploadwg

Implemented direct file upload as alternative import method for WG. For testing and data corrections it is useful to be able to import waterway gauges data directly by uploading a xml file.

author	Sascha Wilde <wilde@sha-bang.de>
date	Thu, 18 Apr 2024 19:23:19 +0200
parents	6270951dda28
children

rev	line source
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2021 by via donau
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "errors"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "fmt"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "time"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	22
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	23 "gemma.intevation.de/gemma/pkg/auth"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24 "gemma.intevation.de/gemma/pkg/common"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "gemma.intevation.de/gemma/pkg/models"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 )
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	28 // StatsUpdate is job for status updates.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29 type StatsUpdate struct {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 models.QueueConfigurationType
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 Name string `json:"name"`
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	34 // StatsUpdateJobKind is the unique name of this import job type.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 const StatsUpdateJobKind JobKind = "statsupdate"
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 type statsUpdateJobCreator struct{}
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 func init() { RegisterJobCreator(StatsUpdateJobKind, statsUpdateJobCreator{}) }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 func (statsUpdateJobCreator) Description() string { return "statsupdate" }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 func (statsUpdateJobCreator) AutoAccept() bool { return true }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 func (statsUpdateJobCreator) Create() Job { return new(StatsUpdate) }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 func (statsUpdateJobCreator) Depends() [2][]string { return [2][]string{{}, {}} }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 func (statsUpdateJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50 return nil
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 // RequiresRoles enforces to be a sys_admin to run this .
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 func (*StatsUpdate) RequiresRoles() auth.Roles { return auth.Roles{"sys_admin"} }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	56 // Description gives a short info about relevant facts of this import.
5565 ade07a3f2cfd Forget to change signature of some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5387 diff changeset	57 func (su *StatsUpdate) Description([]string) (string, error) { return su.Name, nil }
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	59 // CleanUp is an empty implementation.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 func (*StatsUpdate) CleanUp() error { return nil }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	62 // MarshalAttributes implements DB marshaling of this job.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 func (su *StatsUpdate) MarshalAttributes(attrs common.Attributes) error {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 if err := su.QueueConfigurationType.MarshalAttributes(attrs); err != nil {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 return err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 attrs.Set("name", su.Name)
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 return nil
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	71 // UnmarshalAttributes implements DB unmarshaling this job.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 func (su *StatsUpdate) UnmarshalAttributes(attrs common.Attributes) error {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 if err := su.QueueConfigurationType.UnmarshalAttributes(attrs); err != nil {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 return err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76 name, found := attrs.Get("name")
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 if !found {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78 return errors.New("missing 'name' attribute")
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	79 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 su.Name = name
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 return nil
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	82 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	83
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 const loadUpdateStatsScriptSQL = `SELECT script FROM sys_admin.stats_updates WHERE name = $1`
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85
5601 1222b777f51f Made golint finally happy. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5565 diff changeset	86 // Do executes the actual report generation.
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	87 func (su *StatsUpdate) Do(
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 ctx context.Context,
5711 2dd155cc95ec Fix all revive issue (w/o machine generated stuff). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5601 diff changeset	89 _ int64,
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 conn *sql.Conn,
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91 feedback Feedback,
5712 6270951dda28 /interface{}/any/ Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5711 diff changeset	92 ) (any, error) {
5387 8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 start := time.Now()
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 feedback.Info("Running stats update %s.", su.Name)
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	98 tx, err := conn.BeginTx(ctx, nil)
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	99 if err != nil {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	100 return nil, err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 defer tx.Rollback()
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 var script string
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 switch err := tx.QueryRowContext(ctx, loadUpdateStatsScriptSQL, su.Name).Scan(&script); {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 case err == sql.ErrNoRows:
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 return nil, fmt.Errorf("no update script found for '%s'", su.Name)
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 case err != nil:
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109 return nil, err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	110 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	111
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	112 if _, err := tx.ExecContext(ctx, script); err != nil {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	113 return nil, err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	114 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	115
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	116 if err := tx.Commit(); err != nil {
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	117 return nil, err
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	118 }
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	119
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	120 feedback.Info("Running stats update took %v.", time.Since(start))
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	121
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	122 return nil, nil
8e30b926b94d Added an import to run update stats scripts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	123 }

Mercurial > gemma

annotate pkg/imports/statsupdate.go @ 5718:3d497077f888 uploadwg