gemma: pkg/imports/queue.go annotate

annotate pkg/imports/queue.go @ 3654:26e02fa5b992 single-beam

Fixed problem with wrong EPSG code.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 13 Jun 2019 17:07:28 +0200
parents	64324aaeb1fb
children	350a24c92848

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	13
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	19 "encoding/json"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	20 "fmt"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	22 "runtime/debug"
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	23 "sort"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	24 "strings"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "sync"
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	26 "time"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	28 "github.com/jackc/pgx/pgtype"
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	29
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 "gemma.intevation.de/gemma/pkg/auth"
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	31 "gemma.intevation.de/gemma/pkg/common"
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	32 "gemma.intevation.de/gemma/pkg/config"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 )
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	35 type (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	36 // Feedback is passed to the Do method of a Job to log
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	37 // informations, warnings or errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	38 Feedback interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	39 // Info logs informations.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	40 Info(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	41 // Warn logs warnings.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	42 Warn(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	43 // Error logs errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	44 Error(fmt string, args ...interface{})
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	45 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	46
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	47 // UnchangedError may be issued by Do of a Job to indicate
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	48 // That the database has not changed.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	49 UnchangedError string
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	50
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	51 // Job is the central abstraction of an import job
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	52 // run by the import queue.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	53 Job interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	54 // Do is called to do the actual import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	55 // Bind transactions to ctx and conn, please-
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	56 // id is the number of the import job.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	57 // feedback can be used to log the import process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	58 // If no error is return the import is assumed to
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	59 // be successfull. The non-error return value is
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	60 // serialized as a JSON string into the database as
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	61 // a summary to the import to be used by the review process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	62 Do(ctx context.Context, id int64, conn *sql.Conn, feedback Feedback) (interface{}, error)
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	63 // CleanUp is called to clean up ressources hold by the import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	64 // It is called whether the import succeeded or not.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	65 CleanUp() error
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	66 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	67
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	68 // JobKind is the type of an import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	69 // Choose a unique name for every import.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	70 JobKind string
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	71
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	72 // JobCreator is used to bring a job to life as it is stored
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	73 // in pure meta-data form to the database.
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	74 JobCreator interface {
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	75 // Description is the long name of the import.
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	76 Description() string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	77 // Create build the actual job.
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	78 Create() Job
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	79 // Depends returns two lists of ressources locked by this type of import.
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	80 // Imports are run concurrently if they have disjoint sets
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	81 // of dependencies.
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	82 // The first list are locked exclusively.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	83 // The second allows multiple read users but only one writing one.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	84 Depends() [2][]string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	85 // StageDone is called if an import is positively reviewed
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	86 // (state = accepted). This can be used to finalize the imported
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	87 // data to move it e.g from the staging area.
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1327 diff changeset	88 StageDone(context.Context, *sql.Tx, int64) error
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	89 // AutoAccept indicates that imports of this kind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	90 // don't need a review.
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	91 AutoAccept() bool
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	92 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	94 idJob struct {
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	95 id int64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	96 kind JobKind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	97 user string
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	98 waitRetry pgtype.Interval
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	99 trysLeft sql.NullInt64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	100 sendEmail bool
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	101 data string
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	102 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	103 )
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	104
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	105 const (
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	106 pollDuration = time.Second * 10
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	107 runExclusive = -66666
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	108 )
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	109
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	110 type importQueue struct {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	111 signalChan chan struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	112 creatorsMu sync.Mutex
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	113 creators map[JobKind]JobCreator
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	114 usedDeps map[string]int
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	115 }
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	116
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	117 var iqueue = importQueue{
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	118 signalChan: make(chan struct{}),
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	119 creators: map[JobKind]JobCreator{},
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	120 usedDeps: map[string]int{},
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	121 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	122
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	123 var (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	124 // ImportStateNames is a list of the states a job can be in.
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	125 ImportStateNames = []string{
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	126 "queued",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	127 "running",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	128 "failed",
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	129 "unchanged",
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	130 "pending",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	131 "accepted",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	132 "declined",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	133 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	134 )
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	135
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	136 const (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	137 queueUser = "sys_admin"
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	138
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	139 reEnqueueRunningSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	140 UPDATE import.imports SET state = 'queued'::import_state
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	141 WHERE state = 'running'::import_state`
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	142
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	143 insertJobSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	144 INSERT INTO import.imports (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	145 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	146 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	147 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	148 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	149 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	150 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	151 data
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	152 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	153 $1,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	154 COALESCE($2, CURRENT_TIMESTAMP),
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	155 $3,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	156 $4,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	157 $5,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	158 $6,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	159 $7
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	160 ) RETURNING id`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	161
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	162 selectJobSQL = `
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	163 SELECT
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	164 id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	165 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	166 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	167 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	168 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	169 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	170 data
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	171 FROM import.imports
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	172 WHERE
1718 8ddbedf296d7 Re-scheduled imports: be a bit more tolerant about start time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	173 due <= CURRENT_TIMESTAMP + interval '5 seconds' AND
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	174 state = 'queued'::import_state AND enqueued IN (
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	175 SELECT min(enqueued)
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	176 FROM import.imports
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	177 WHERE state = 'queued'::import_state AND
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	178 kind = ANY($1))
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	179 LIMIT 1`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	180
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	181 updateStateSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	182 UPDATE import.imports SET state = $1::import_state
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	183 WHERE id = $2`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	184
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	185 updateStateSummarySQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	186 UPDATE import.imports SET
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	187 state = $1::import_state,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	188 summary = $2
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	189 WHERE id = $3`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	190
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	191 logMessageSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	192 INSERT INTO import.import_logs (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	193 import_id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	194 kind,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	195 msg
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	196 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	197 $1,
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	198 $2::log_type,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	199 $3
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	200 )`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	201 )
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	202
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	203 func init() {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	204 go iqueue.importLoop()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	205 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	206
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	207 // Error makes UnchangedError an error.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	208 func (ue UnchangedError) Error() string {
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	209 return string(ue)
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	210 }
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	211
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	212 func (q *importQueue) registerJobCreator(kind JobKind, jc JobCreator) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	213 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	214 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	215 q.creators[kind] = jc
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	218 // FindJobCreator looks up a JobCreator in the global import queue.
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	219 func FindJobCreator(kind JobKind) JobCreator {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	220 return iqueue.jobCreator(kind)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	221 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	222
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	223 // ImportKindNames is a list of the names of the imports the
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	224 // global import queue supports.
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	225 func ImportKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	226 return iqueue.importKindNames()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	227 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	228
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	229 // LogImportKindNames logs a list of importer types registered
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	230 // to the global import queue.
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	231 func LogImportKindNames() {
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	232 kinds := ImportKindNames()
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	233 sort.Strings(kinds)
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	234 log.Printf("info: registered import kinds: %s",
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	235 strings.Join(kinds, ", "))
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	236 }
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	237
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1657 diff changeset	238 // HasImportKindName checks if the import queue supports a given kind.
1627 b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	239 func HasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	240 return iqueue.hasImportKindName(kind)
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	241 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	242
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	243 //
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	244 func (q *importQueue) hasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	245 q.creatorsMu.Lock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	246 defer q.creatorsMu.Unlock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	247 return q.creators[JobKind(kind)] != nil
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	248 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	249
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	250 // RegisterJobCreator adds a JobCreator to the global import queue.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	251 // This a good candidate to be called in a init function for
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	252 // a particular JobCreator.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	253 func RegisterJobCreator(kind JobKind, jc JobCreator) {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	254 iqueue.registerJobCreator(kind, jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	255 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	256
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	257 func (q *importQueue) importKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	258 q.creatorsMu.Lock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	259 defer q.creatorsMu.Unlock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	260 names := make([]string, len(q.creators))
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	261 var i int
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	262 for kind := range q.creators {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	263 names[i] = string(kind)
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	264 i++
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	265 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	266 // XXX: Consider using sort.Strings to make output deterministic.
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	267 return names
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	268 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	269
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	270 func (idj *idJob) nextRetry(feedback Feedback) bool {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	271 switch {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	272 case idj.waitRetry.Status != pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	273 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	274 case idj.waitRetry.Status == pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	275 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	276 case idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	277 if idj.trysLeft.Int64 < 1 {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	278 feedback.Warn("import should be retried, but no retrys left")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	279 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	280 idj.trysLeft.Int64--
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	281 feedback.Info("import failed but will be retried")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	282 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	283 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	284 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	285 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	286 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	287
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	288 func (idj *idJob) nextDue() time.Time {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	289 now := time.Now()
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	290 if idj.waitRetry.Status == pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	291 var d time.Duration
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	292 if err := idj.waitRetry.AssignTo(&d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	293 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	294 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	295 now = now.Add(d)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	296 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	297 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	298 return now
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	299 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	300
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	301 func (idj idJob) trysLeftPointer() int {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	302 if !idj.trysLeft.Valid {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	303 return nil
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	304 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	305 t := int(idj.trysLeft.Int64)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	306 return &t
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	307 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	308
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	309 func (idj idJob) waitRetryPointer() time.Duration {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	310 if idj.waitRetry.Status != pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	311 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	312 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	313 d := new(time.Duration)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	314 if err := idj.waitRetry.AssignTo(d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	315 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	316 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	317 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	318 return d
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	319 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	320
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	321 func (q *importQueue) lockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	322 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	323 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	324 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	325 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	326 q.usedDeps[d] = runExclusive
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	327 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	328 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	329 q.usedDeps[d]++
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	330 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	331 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	332
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	333 func (q *importQueue) unlockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	334 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	335 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	336 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	337 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	338 q.usedDeps[d] = 0
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	339 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	340 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	341 q.usedDeps[d]--
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	342 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	343 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	344
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	345 func (q *importQueue) jobCreator(kind JobKind) JobCreator {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	346 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	347 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	348 return q.creators[kind]
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	349 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	350
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	351 func (q *importQueue) addJob(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	352 kind JobKind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	353 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	354 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	355 waitRetry *time.Duration,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	356 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	357 sendEmail bool,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	358 data string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	359 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	360
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	361 var id int64
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	362 if due.IsZero() {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	363 due = time.Now()
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	364 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	365
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	366 var tl sql.NullInt64
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	367 if trysLeft != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	368 tl = sql.NullInt64{Int64: int64(*trysLeft), Valid: true}
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	369 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	370
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	371 var wr pgtype.Interval
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	372 if waitRetry != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	373 if err := wr.Set(*waitRetry); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	374 return 0, err
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	375 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	376 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	377 wr = pgtype.Interval{Status: pgtype.Null}
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	378 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	379
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	380 ctx := context.Background()
1798 40cbfd268aa9 Row level security for import jobs Tom Gottfried <tom@intevation.de> parents: 1760 diff changeset	381 err := auth.RunAs(ctx, user, func(conn *sql.Conn) error {
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	382 return conn.QueryRowContext(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	383 ctx,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	384 insertJobSQL,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	385 string(kind),
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	386 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	387 tl,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	388 &wr,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	389 user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	390 sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	391 data).Scan(&id)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	392 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	393 if err == nil {
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	394 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	395 case q.signalChan <- struct{}{}:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	396 default:
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	397 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	398 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	399 return id, err
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	400 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	401
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	402 // AddJob adds a job to the global import queue to be executed
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	403 // as soon as possible after due.
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	404 // This is gone in a separate Go routine
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	405 // so this will not block.
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	406 func AddJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	407 kind JobKind,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	408 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	409 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	410 waitRetry *time.Duration,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	411 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	412 sendEmail bool,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	413 data string,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	414 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	415 return iqueue.addJob(
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	416 kind,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	417 due,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	418 trysLeft,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	419 waitRetry,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	420 user,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	421 sendEmail,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	422 data)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	423 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	424
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	425 type logFeedback int64
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	426
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	427 func (lf logFeedback) log(kind, format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	428 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	429 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	430 _, err := conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	431 ctx, logMessageSQL, int64(lf), kind, fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	432 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	433 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	434 if err != nil {
1760 22148eb0f986 More on harmonizing logging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1758 diff changeset	435 log.Printf("error: logging failed: %v\n", err)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	436 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	437 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	438
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	439 func (lf logFeedback) Info(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	440 lf.log("info", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	441 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	442
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	443 func (lf logFeedback) Warn(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	444 lf.log("warn", format, args...)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	445 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	446
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	447 func (lf logFeedback) Error(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	448 lf.log("error", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	449 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	450
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	451 func survive(fn func() error) func() error {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	452 return func() (err error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	453 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	454 if p := recover(); p != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	455 err = fmt.Errorf("%v: %s", p, string(debug.Stack()))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	456 }
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	457 }()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	458 return fn()
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	459 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	460 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	461
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	462 func reEnqueueRunning() error {
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	463 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	464 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	465 _, err := conn.ExecContext(ctx, reEnqueueRunningSQL)
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	466 return err
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	467 })
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	468 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	469
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	470 func (q importQueue) fetchJob() (idJob, error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	471
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	472 var which []string
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	473
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	474 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	475 nextCreator:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	476 for kind, jc := range q.creators {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	477 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	478 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	479 if q.usedDeps[d] != 0 {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	480 continue nextCreator
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	481 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	482 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	483 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	484 if q.usedDeps[d] == runExclusive {
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	485 continue nextCreator
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	486 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	487 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	488 which = append(which, string(kind))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	489 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	490 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	491
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	492 if len(which) == 0 {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	493 return nil, sql.ErrNoRows
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	494 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	495
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	496 var kinds pgtype.TextArray
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	497 if err := kinds.Set(which); err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	498 return nil, err
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	499 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	500
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	501 var ji idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	502 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	503 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	504 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	505 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	506 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	507 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	508 defer tx.Rollback()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	509 if err = tx.QueryRowContext(ctx, selectJobSQL, &kinds).Scan(
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	510 &ji.id,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	511 &ji.kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	512 &ji.trysLeft,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	513 &ji.waitRetry,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	514 &ji.user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	515 &ji.sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	516 &ji.data,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	517 ); err != nil {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	518 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	519 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	520 _, err = tx.ExecContext(ctx, updateStateSQL, "running", ji.id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	521 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	522 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	523 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	524 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	525 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	526 switch {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	527 case err == sql.ErrNoRows:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	528 return nil, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	529 case err != nil:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	530 return nil, err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	531 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	532 return &ji, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	533 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	534
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	535 func tryHardToStoreState(ctx context.Context, fn func(*sql.Conn) error) error {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	536 // As it is important to keep the persistent model
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	537 // in sync with the in-memory model try harder to store
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	538 // the state.
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	539 const maxTries = 10
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	540 var sleep = time.Second
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	541
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	542 for try := 1; ; try++ {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	543 var err error
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	544 if err = auth.RunAs(ctx, queueUser, fn); err == nil \|\| try == maxTries {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	545 return err
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	546 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	547 log.Printf("warn: [try %d/%d] Storing state failed: %v (try again in %s).\n",
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	548 try, maxTries, err, sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	549
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	550 time.Sleep(sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	551 if sleep < time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	552 if sleep *= 2; sleep > time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	553 sleep = time.Minute
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	554 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	555 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	556 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	557 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	558
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	559 func updateStateSummary(
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	560 ctx context.Context,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	561 id int64,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	562 state string,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	563 summary interface{},
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	564 ) error {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	565 var s sql.NullString
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	566 if summary != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	567 var b strings.Builder
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	568 if err := json.NewEncoder(&b).Encode(summary); err != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	569 return err
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	570 }
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	571 s = sql.NullString{String: b.String(), Valid: true}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	572 }
2823 b150e5b37afe Import queue: As it is important to keep the in-memory and the persistent model in sync try harder to store the final state change if it fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2801 diff changeset	573
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	574 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	575 _, err := conn.ExecContext(ctx, updateStateSummarySQL, state, s, id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	576 return err
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	577 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	578 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	579
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	580 func errorAndFail(id int64, format string, args ...interface{}) error {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	581 ctx := context.Background()
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	582 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	583 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	584 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	585 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	586 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	587 defer tx.Rollback()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	588 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	589 ctx, logMessageSQL, id, "error", fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	590 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	591 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	592 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	593 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	594 ctx, updateStateSQL, "failed", id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	595 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	596 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	597 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	598 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	599 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	600 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	601
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	602 func (q *importQueue) importLoop() {
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	603 config.WaitReady()
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	604 // re-enqueue the jobs that are in state running.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	605 // They where in progess when the server went down.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	606 if err := reEnqueueRunning(); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	607 log.Printf("error: re-enqueuing failed: %v", err)
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	608 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	609
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	610 for {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	611 var idj *idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	612 var err error
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	613
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	614 for {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	615 if idj, err = q.fetchJob(); err != nil && err != sql.ErrNoRows {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	616 log.Printf("error: db: %v\n", err)
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	617 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	618 if idj != nil {
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	619 break
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	620 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	621 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	622 case <-q.signalChan:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	623 case <-time.After(pollDuration):
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	624 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	625 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	626
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	627 log.Printf("info: starting import #%d\n", idj.id)
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	628
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	629 jc := q.jobCreator(idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	630 if jc == nil {
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	631 errorAndFail(idj.id, "no creator for kind '%s' found", idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	632 continue
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	633 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	634
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	635 // Lock dependencies.
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	636 q.lockDependencies(jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	637
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	638 go func(jc JobCreator, idj *idJob) {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	639
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	640 // Unlock the dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	641 defer func() {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	642 q.unlockDependencies(jc)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	643 select {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	644 case q.signalChan <- struct{}{}:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	645 default:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	646 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	647 }()
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	648
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	649 job := jc.Create()
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	650 if err := common.FromJSONString(idj.data, job); err != nil {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	651 errorAndFail(idj.id, "failed to create job for import #%d: %v",
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	652 idj.id, err)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	653 return
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	654 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	655
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	656 feedback := logFeedback(idj.id)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	657
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1136 diff changeset	658 feedback.Info("import #%d started", idj.id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	659
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	660 ctx := context.Background()
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	661 var summary interface{}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	662
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	663 errDo := survive(func() error {
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	664 return auth.RunAs(ctx, idj.user,
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	665 func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	666 var err error
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	667 summary, err = job.Do(ctx, idj.id, conn, feedback)
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	668 return err
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	669 })
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	670 })()
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	671
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	672 var unchanged, retry bool
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	673 if v, ok := errDo.(UnchangedError); ok {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	674 feedback.Info("unchanged: %s", v.Error())
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	675 unchanged = true
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	676 } else if errDo != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	677 feedback.Error("error in import: %v", errDo)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	678 retry = idj.nextRetry(feedback)
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	679 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	680
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	681 var errCleanup error
2801 054f5d61452d Import queue: Cleanup up debris after an import in all cases if we are not going to retry it. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	682 if !retry { // cleanup debris
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	683 if errCleanup = survive(job.CleanUp)(); errCleanup != nil {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	684 feedback.Error("error cleanup: %v", errCleanup)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	685 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	686 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	687
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	688 var state string
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	689 switch {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	690 case unchanged:
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	691 state = "unchanged"
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	692 case errDo != nil \|\| errCleanup != nil:
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	693 state = "failed"
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	694 case jc.AutoAccept():
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	695 state = "accepted"
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	696 default:
1190 e3de65179889 The imort queue has now six states: Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	697 state = "pending"
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	698 }
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	699 if err := updateStateSummary(ctx, idj.id, state, summary); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	700 log.Printf("error: setting state of job %d failed: %v\n", idj.id, err)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	701 }
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	702 log.Printf("info: import #%d finished: %s\n", idj.id, state)
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	703 if idj.sendEmail {
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	704 go sendNotificationMail(idj.user, jc.Description(), state, idj.id)
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	705 }
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	706
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	707 if retry {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	708 nid, err := q.addJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	709 idj.kind,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	710 idj.nextDue(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	711 idj.trysLeftPointer(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	712 idj.waitRetryPointer(),
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	713 idj.user, idj.sendEmail,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	714 idj.data)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	715 if err != nil {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	716 log.Printf("error: retry enqueue failed: %v\n", err)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	717 } else {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	718 log.Printf("info: re-enqueued job with id %d\n", nid)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	719 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	720 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	721 }(jc, idj)
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	722 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	723 }

Mercurial > gemma

annotate pkg/imports/queue.go @ 3654:26e02fa5b992 single-beam