gemma: pkg/imports/queue.go annotate

annotate pkg/imports/queue.go @ 3678:8f58851927c0

client: make layer factory only return new layer config for individual maps instead of each time it is invoked. The purpose of the factory was to support multiple maps with individual layers. But returning a new config each time it is invoked leads to bugs that rely on the layer's state. Now this factory reuses the same objects it created before, per map.

author	Markus Kottlaender <markus@intevation.de>
date	Mon, 17 Jun 2019 17:31:35 +0200
parents	64324aaeb1fb
children	350a24c92848

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	13
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	19 "encoding/json"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	20 "fmt"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	22 "runtime/debug"
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	23 "sort"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	24 "strings"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "sync"
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	26 "time"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	28 "github.com/jackc/pgx/pgtype"
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	29
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 "gemma.intevation.de/gemma/pkg/auth"
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	31 "gemma.intevation.de/gemma/pkg/common"
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	32 "gemma.intevation.de/gemma/pkg/config"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 )
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	35 type (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	36 // Feedback is passed to the Do method of a Job to log
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	37 // informations, warnings or errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	38 Feedback interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	39 // Info logs informations.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	40 Info(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	41 // Warn logs warnings.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	42 Warn(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	43 // Error logs errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	44 Error(fmt string, args ...interface{})
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	45 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	46
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	47 // UnchangedError may be issued by Do of a Job to indicate
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	48 // That the database has not changed.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	49 UnchangedError string
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	50
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	51 // Job is the central abstraction of an import job
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	52 // run by the import queue.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	53 Job interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	54 // Do is called to do the actual import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	55 // Bind transactions to ctx and conn, please-
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	56 // id is the number of the import job.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	57 // feedback can be used to log the import process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	58 // If no error is return the import is assumed to
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	59 // be successfull. The non-error return value is
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	60 // serialized as a JSON string into the database as
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	61 // a summary to the import to be used by the review process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	62 Do(ctx context.Context, id int64, conn *sql.Conn, feedback Feedback) (interface{}, error)
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	63 // CleanUp is called to clean up ressources hold by the import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	64 // It is called whether the import succeeded or not.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	65 CleanUp() error
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	66 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	67
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	68 // JobKind is the type of an import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	69 // Choose a unique name for every import.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	70 JobKind string
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	71
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	72 // JobCreator is used to bring a job to life as it is stored
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	73 // in pure meta-data form to the database.
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	74 JobCreator interface {
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	75 // Description is the long name of the import.
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	76 Description() string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	77 // Create build the actual job.
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	78 Create() Job
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	79 // Depends returns two lists of ressources locked by this type of import.
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	80 // Imports are run concurrently if they have disjoint sets
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	81 // of dependencies.
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	82 // The first list are locked exclusively.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	83 // The second allows multiple read users but only one writing one.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	84 Depends() [2][]string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	85 // StageDone is called if an import is positively reviewed
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	86 // (state = accepted). This can be used to finalize the imported
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	87 // data to move it e.g from the staging area.
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1327 diff changeset	88 StageDone(context.Context, *sql.Tx, int64) error
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	89 // AutoAccept indicates that imports of this kind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	90 // don't need a review.
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	91 AutoAccept() bool
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	92 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	94 idJob struct {
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	95 id int64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	96 kind JobKind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	97 user string
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	98 waitRetry pgtype.Interval
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	99 trysLeft sql.NullInt64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	100 sendEmail bool
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	101 data string
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	102 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	103 )
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	104
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	105 const (
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	106 pollDuration = time.Second * 10
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	107 runExclusive = -66666
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	108 )
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	109
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	110 type importQueue struct {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	111 signalChan chan struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	112 creatorsMu sync.Mutex
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	113 creators map[JobKind]JobCreator
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	114 usedDeps map[string]int
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	115 }
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	116
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	117 var iqueue = importQueue{
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	118 signalChan: make(chan struct{}),
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	119 creators: map[JobKind]JobCreator{},
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	120 usedDeps: map[string]int{},
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	121 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	122
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	123 var (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	124 // ImportStateNames is a list of the states a job can be in.
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	125 ImportStateNames = []string{
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	126 "queued",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	127 "running",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	128 "failed",
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	129 "unchanged",
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	130 "pending",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	131 "accepted",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	132 "declined",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	133 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	134 )
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	135
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	136 const (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	137 queueUser = "sys_admin"
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	138
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	139 reEnqueueRunningSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	140 UPDATE import.imports SET state = 'queued'::import_state
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	141 WHERE state = 'running'::import_state`
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	142
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	143 insertJobSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	144 INSERT INTO import.imports (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	145 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	146 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	147 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	148 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	149 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	150 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	151 data
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	152 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	153 $1,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	154 COALESCE($2, CURRENT_TIMESTAMP),
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	155 $3,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	156 $4,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	157 $5,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	158 $6,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	159 $7
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	160 ) RETURNING id`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	161
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	162 selectJobSQL = `
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	163 SELECT
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	164 id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	165 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	166 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	167 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	168 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	169 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	170 data
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	171 FROM import.imports
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	172 WHERE
1718 8ddbedf296d7 Re-scheduled imports: be a bit more tolerant about start time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	173 due <= CURRENT_TIMESTAMP + interval '5 seconds' AND
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	174 state = 'queued'::import_state AND enqueued IN (
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	175 SELECT min(enqueued)
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	176 FROM import.imports
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	177 WHERE state = 'queued'::import_state AND
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	178 kind = ANY($1))
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	179 LIMIT 1`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	180
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	181 updateStateSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	182 UPDATE import.imports SET state = $1::import_state
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	183 WHERE id = $2`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	184
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	185 updateStateSummarySQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	186 UPDATE import.imports SET
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	187 state = $1::import_state,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	188 summary = $2
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	189 WHERE id = $3`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	190
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	191 logMessageSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	192 INSERT INTO import.import_logs (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	193 import_id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	194 kind,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	195 msg
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	196 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	197 $1,
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	198 $2::log_type,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	199 $3
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	200 )`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	201 )
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	202
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	203 func init() {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	204 go iqueue.importLoop()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	205 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	206
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	207 // Error makes UnchangedError an error.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	208 func (ue UnchangedError) Error() string {
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	209 return string(ue)
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	210 }
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	211
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	212 func (q *importQueue) registerJobCreator(kind JobKind, jc JobCreator) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	213 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	214 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	215 q.creators[kind] = jc
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	218 // FindJobCreator looks up a JobCreator in the global import queue.
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	219 func FindJobCreator(kind JobKind) JobCreator {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	220 return iqueue.jobCreator(kind)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	221 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	222
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	223 // ImportKindNames is a list of the names of the imports the
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	224 // global import queue supports.
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	225 func ImportKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	226 return iqueue.importKindNames()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	227 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	228
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	229 // LogImportKindNames logs a list of importer types registered
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	230 // to the global import queue.
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	231 func LogImportKindNames() {
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	232 kinds := ImportKindNames()
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	233 sort.Strings(kinds)
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	234 log.Printf("info: registered import kinds: %s",
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	235 strings.Join(kinds, ", "))
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	236 }
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	237
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1657 diff changeset	238 // HasImportKindName checks if the import queue supports a given kind.
1627 b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	239 func HasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	240 return iqueue.hasImportKindName(kind)
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	241 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	242
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	243 //
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	244 func (q *importQueue) hasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	245 q.creatorsMu.Lock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	246 defer q.creatorsMu.Unlock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	247 return q.creators[JobKind(kind)] != nil
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	248 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	249
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	250 // RegisterJobCreator adds a JobCreator to the global import queue.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	251 // This a good candidate to be called in a init function for
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	252 // a particular JobCreator.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	253 func RegisterJobCreator(kind JobKind, jc JobCreator) {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	254 iqueue.registerJobCreator(kind, jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	255 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	256
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	257 func (q *importQueue) importKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	258 q.creatorsMu.Lock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	259 defer q.creatorsMu.Unlock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	260 names := make([]string, len(q.creators))
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	261 var i int
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	262 for kind := range q.creators {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	263 names[i] = string(kind)
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	264 i++
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	265 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	266 // XXX: Consider using sort.Strings to make output deterministic.
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	267 return names
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	268 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	269
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	270 func (idj *idJob) nextRetry(feedback Feedback) bool {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	271 switch {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	272 case idj.waitRetry.Status != pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	273 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	274 case idj.waitRetry.Status == pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	275 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	276 case idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	277 if idj.trysLeft.Int64 < 1 {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	278 feedback.Warn("import should be retried, but no retrys left")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	279 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	280 idj.trysLeft.Int64--
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	281 feedback.Info("import failed but will be retried")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	282 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	283 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	284 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	285 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	286 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	287
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	288 func (idj *idJob) nextDue() time.Time {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	289 now := time.Now()
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	290 if idj.waitRetry.Status == pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	291 var d time.Duration
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	292 if err := idj.waitRetry.AssignTo(&d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	293 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	294 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	295 now = now.Add(d)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	296 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	297 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	298 return now
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	299 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	300
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	301 func (idj idJob) trysLeftPointer() int {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	302 if !idj.trysLeft.Valid {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	303 return nil
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	304 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	305 t := int(idj.trysLeft.Int64)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	306 return &t
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	307 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	308
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	309 func (idj idJob) waitRetryPointer() time.Duration {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	310 if idj.waitRetry.Status != pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	311 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	312 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	313 d := new(time.Duration)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	314 if err := idj.waitRetry.AssignTo(d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	315 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	316 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	317 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	318 return d
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	319 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	320
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	321 func (q *importQueue) lockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	322 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	323 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	324 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	325 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	326 q.usedDeps[d] = runExclusive
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	327 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	328 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	329 q.usedDeps[d]++
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	330 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	331 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	332
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	333 func (q *importQueue) unlockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	334 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	335 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	336 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	337 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	338 q.usedDeps[d] = 0
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	339 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	340 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	341 q.usedDeps[d]--
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	342 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	343 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	344
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	345 func (q *importQueue) jobCreator(kind JobKind) JobCreator {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	346 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	347 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	348 return q.creators[kind]
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	349 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	350
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	351 func (q *importQueue) addJob(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	352 kind JobKind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	353 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	354 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	355 waitRetry *time.Duration,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	356 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	357 sendEmail bool,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	358 data string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	359 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	360
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	361 var id int64
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	362 if due.IsZero() {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	363 due = time.Now()
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	364 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	365
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	366 var tl sql.NullInt64
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	367 if trysLeft != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	368 tl = sql.NullInt64{Int64: int64(*trysLeft), Valid: true}
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	369 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	370
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	371 var wr pgtype.Interval
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	372 if waitRetry != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	373 if err := wr.Set(*waitRetry); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	374 return 0, err
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	375 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	376 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	377 wr = pgtype.Interval{Status: pgtype.Null}
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	378 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	379
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	380 ctx := context.Background()
1798 40cbfd268aa9 Row level security for import jobs Tom Gottfried <tom@intevation.de> parents: 1760 diff changeset	381 err := auth.RunAs(ctx, user, func(conn *sql.Conn) error {
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	382 return conn.QueryRowContext(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	383 ctx,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	384 insertJobSQL,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	385 string(kind),
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	386 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	387 tl,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	388 &wr,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	389 user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	390 sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	391 data).Scan(&id)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	392 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	393 if err == nil {
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	394 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	395 case q.signalChan <- struct{}{}:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	396 default:
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	397 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	398 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	399 return id, err
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	400 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	401
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	402 // AddJob adds a job to the global import queue to be executed
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	403 // as soon as possible after due.
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	404 // This is gone in a separate Go routine
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	405 // so this will not block.
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	406 func AddJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	407 kind JobKind,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	408 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	409 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	410 waitRetry *time.Duration,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	411 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	412 sendEmail bool,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	413 data string,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	414 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	415 return iqueue.addJob(
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	416 kind,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	417 due,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	418 trysLeft,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	419 waitRetry,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	420 user,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	421 sendEmail,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	422 data)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	423 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	424
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	425 type logFeedback int64
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	426
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	427 func (lf logFeedback) log(kind, format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	428 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	429 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	430 _, err := conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	431 ctx, logMessageSQL, int64(lf), kind, fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	432 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	433 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	434 if err != nil {
1760 22148eb0f986 More on harmonizing logging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1758 diff changeset	435 log.Printf("error: logging failed: %v\n", err)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	436 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	437 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	438
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	439 func (lf logFeedback) Info(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	440 lf.log("info", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	441 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	442
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	443 func (lf logFeedback) Warn(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	444 lf.log("warn", format, args...)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	445 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	446
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	447 func (lf logFeedback) Error(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	448 lf.log("error", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	449 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	450
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	451 func survive(fn func() error) func() error {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	452 return func() (err error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	453 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	454 if p := recover(); p != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	455 err = fmt.Errorf("%v: %s", p, string(debug.Stack()))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	456 }
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	457 }()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	458 return fn()
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	459 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	460 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	461
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	462 func reEnqueueRunning() error {
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	463 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	464 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	465 _, err := conn.ExecContext(ctx, reEnqueueRunningSQL)
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	466 return err
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	467 })
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	468 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	469
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	470 func (q importQueue) fetchJob() (idJob, error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	471
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	472 var which []string
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	473
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	474 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	475 nextCreator:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	476 for kind, jc := range q.creators {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	477 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	478 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	479 if q.usedDeps[d] != 0 {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	480 continue nextCreator
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	481 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	482 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	483 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	484 if q.usedDeps[d] == runExclusive {
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	485 continue nextCreator
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	486 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	487 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	488 which = append(which, string(kind))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	489 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	490 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	491
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	492 if len(which) == 0 {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	493 return nil, sql.ErrNoRows
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	494 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	495
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	496 var kinds pgtype.TextArray
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	497 if err := kinds.Set(which); err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	498 return nil, err
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	499 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	500
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	501 var ji idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	502 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	503 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	504 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	505 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	506 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	507 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	508 defer tx.Rollback()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	509 if err = tx.QueryRowContext(ctx, selectJobSQL, &kinds).Scan(
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	510 &ji.id,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	511 &ji.kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	512 &ji.trysLeft,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	513 &ji.waitRetry,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	514 &ji.user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	515 &ji.sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	516 &ji.data,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	517 ); err != nil {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	518 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	519 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	520 _, err = tx.ExecContext(ctx, updateStateSQL, "running", ji.id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	521 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	522 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	523 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	524 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	525 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	526 switch {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	527 case err == sql.ErrNoRows:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	528 return nil, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	529 case err != nil:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	530 return nil, err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	531 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	532 return &ji, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	533 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	534
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	535 func tryHardToStoreState(ctx context.Context, fn func(*sql.Conn) error) error {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	536 // As it is important to keep the persistent model
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	537 // in sync with the in-memory model try harder to store
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	538 // the state.
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	539 const maxTries = 10
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	540 var sleep = time.Second
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	541
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	542 for try := 1; ; try++ {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	543 var err error
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	544 if err = auth.RunAs(ctx, queueUser, fn); err == nil \|\| try == maxTries {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	545 return err
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	546 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	547 log.Printf("warn: [try %d/%d] Storing state failed: %v (try again in %s).\n",
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	548 try, maxTries, err, sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	549
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	550 time.Sleep(sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	551 if sleep < time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	552 if sleep *= 2; sleep > time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	553 sleep = time.Minute
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	554 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	555 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	556 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	557 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	558
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	559 func updateStateSummary(
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	560 ctx context.Context,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	561 id int64,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	562 state string,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	563 summary interface{},
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	564 ) error {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	565 var s sql.NullString
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	566 if summary != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	567 var b strings.Builder
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	568 if err := json.NewEncoder(&b).Encode(summary); err != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	569 return err
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	570 }
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	571 s = sql.NullString{String: b.String(), Valid: true}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	572 }
2823 b150e5b37afe Import queue: As it is important to keep the in-memory and the persistent model in sync try harder to store the final state change if it fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2801 diff changeset	573
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	574 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	575 _, err := conn.ExecContext(ctx, updateStateSummarySQL, state, s, id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	576 return err
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	577 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	578 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	579
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	580 func errorAndFail(id int64, format string, args ...interface{}) error {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	581 ctx := context.Background()
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	582 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	583 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	584 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	585 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	586 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	587 defer tx.Rollback()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	588 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	589 ctx, logMessageSQL, id, "error", fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	590 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	591 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	592 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	593 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	594 ctx, updateStateSQL, "failed", id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	595 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	596 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	597 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	598 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	599 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	600 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	601
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	602 func (q *importQueue) importLoop() {
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	603 config.WaitReady()
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	604 // re-enqueue the jobs that are in state running.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	605 // They where in progess when the server went down.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	606 if err := reEnqueueRunning(); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	607 log.Printf("error: re-enqueuing failed: %v", err)
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	608 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	609
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	610 for {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	611 var idj *idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	612 var err error
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	613
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	614 for {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	615 if idj, err = q.fetchJob(); err != nil && err != sql.ErrNoRows {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	616 log.Printf("error: db: %v\n", err)
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	617 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	618 if idj != nil {
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	619 break
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	620 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	621 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	622 case <-q.signalChan:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	623 case <-time.After(pollDuration):
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	624 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	625 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	626
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	627 log.Printf("info: starting import #%d\n", idj.id)
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	628
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	629 jc := q.jobCreator(idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	630 if jc == nil {
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	631 errorAndFail(idj.id, "no creator for kind '%s' found", idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	632 continue
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	633 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	634
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	635 // Lock dependencies.
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	636 q.lockDependencies(jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	637
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	638 go func(jc JobCreator, idj *idJob) {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	639
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	640 // Unlock the dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	641 defer func() {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	642 q.unlockDependencies(jc)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	643 select {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	644 case q.signalChan <- struct{}{}:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	645 default:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	646 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	647 }()
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	648
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	649 job := jc.Create()
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	650 if err := common.FromJSONString(idj.data, job); err != nil {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	651 errorAndFail(idj.id, "failed to create job for import #%d: %v",
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	652 idj.id, err)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	653 return
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	654 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	655
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	656 feedback := logFeedback(idj.id)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	657
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1136 diff changeset	658 feedback.Info("import #%d started", idj.id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	659
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	660 ctx := context.Background()
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	661 var summary interface{}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	662
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	663 errDo := survive(func() error {
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	664 return auth.RunAs(ctx, idj.user,
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	665 func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	666 var err error
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	667 summary, err = job.Do(ctx, idj.id, conn, feedback)
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	668 return err
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	669 })
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	670 })()
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	671
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	672 var unchanged, retry bool
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	673 if v, ok := errDo.(UnchangedError); ok {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	674 feedback.Info("unchanged: %s", v.Error())
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	675 unchanged = true
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	676 } else if errDo != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	677 feedback.Error("error in import: %v", errDo)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	678 retry = idj.nextRetry(feedback)
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	679 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	680
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	681 var errCleanup error
2801 054f5d61452d Import queue: Cleanup up debris after an import in all cases if we are not going to retry it. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	682 if !retry { // cleanup debris
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	683 if errCleanup = survive(job.CleanUp)(); errCleanup != nil {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	684 feedback.Error("error cleanup: %v", errCleanup)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	685 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	686 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	687
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	688 var state string
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	689 switch {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	690 case unchanged:
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	691 state = "unchanged"
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	692 case errDo != nil \|\| errCleanup != nil:
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	693 state = "failed"
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	694 case jc.AutoAccept():
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	695 state = "accepted"
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	696 default:
1190 e3de65179889 The imort queue has now six states: Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	697 state = "pending"
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	698 }
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	699 if err := updateStateSummary(ctx, idj.id, state, summary); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	700 log.Printf("error: setting state of job %d failed: %v\n", idj.id, err)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	701 }
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	702 log.Printf("info: import #%d finished: %s\n", idj.id, state)
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	703 if idj.sendEmail {
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	704 go sendNotificationMail(idj.user, jc.Description(), state, idj.id)
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	705 }
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	706
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	707 if retry {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	708 nid, err := q.addJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	709 idj.kind,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	710 idj.nextDue(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	711 idj.trysLeftPointer(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	712 idj.waitRetryPointer(),
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	713 idj.user, idj.sendEmail,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	714 idj.data)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	715 if err != nil {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	716 log.Printf("error: retry enqueue failed: %v\n", err)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	717 } else {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	718 log.Printf("info: re-enqueued job with id %d\n", nid)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	719 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	720 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	721 }(jc, idj)
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	722 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	723 }

Mercurial > gemma

annotate pkg/imports/queue.go @ 3678:8f58851927c0