Mercurial > gemma

annotate pkg/imports/queue.go @ 1751:c3a6aaf926c3

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Import queue: Harmonized logging output.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Thu, 10 Jan 2019 15:15:31 +0100
parents 82a36eaf2366
children 807569b08513
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1017
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
3 //
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
6 //
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
10 //
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
11 // Author(s):
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1010
diff changeset
13
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package imports
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "context"
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
19 "encoding/json"
992
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
20 "fmt"
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "log"
992
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
22 "runtime/debug"
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
23 "strings"
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "sync"
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
25 "time"
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
27 "github.com/jackc/pgx/pgtype"
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
28
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/auth"
1000
14425e35e3c2 Wait with start of import queue until configuration is fully loaded.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 998
diff changeset
30 "gemma.intevation.de/gemma/pkg/config"
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 )
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
33 type (
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
34 // Feedback is passed to the Do method of a Job to log
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
35 // informations, warnings or errors.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
36 Feedback interface {
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
37 // Info logs informations.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
38 Info(fmt string, args ...interface{})
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
39 // Warn logs warnings.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
40 Warn(fmt string, args ...interface{})
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
41 // Error logs errors.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
42 Error(fmt string, args ...interface{})
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
43 }
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
44
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
45 // RetryError is an error type to signal that
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
46 // the import should be tried again.
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
47 RetryError struct {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
48 // Message is the error message.
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
49 Message string
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
50 // When is the new scheduled execution time.
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
51 When time.Time
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
52 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
53
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
54 // Job is the central abstraction of an import job
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
55 // run by the import queue.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
56 Job interface {
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
57 // Do is called to do the actual import.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
58 // Bind transactions to ctx and conn, please-
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
59 // id is the number of the import job.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
60 // feedback can be used to log the import process.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
61 // If no error is return the import is assumed to
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
62 // be successfull. The non-error return value is
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
63 // serialized as a JSON string into the database as
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
64 // a summary to the import to be used by the review process.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
65 Do(ctx context.Context, id int64, conn *sql.Conn, feedback Feedback) (interface{}, error)
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
66 // CleanUp is called to clean up ressources hold by the import.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
67 // It is called whether the import succeeded or not.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
68 CleanUp() error
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
69 }
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
70
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
71 // JobKind is the type of an import.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
72 // Choose a unique name for every import.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
73 JobKind string
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
74
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
75 // JobCreator is used to bring a job to life as it is stored
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
76 // in pure meta-data form to the database.
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
77 JobCreator interface {
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1642
diff changeset
78 // Description is the long name of the import.
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1642
diff changeset
79 Description() string
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
80 // Create build the actual job.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
81 // kind is the name of the import type.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
82 // data is a free form string to pass arguments to the creation
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
83 // process. This is useful to tell e.g. where to find data
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
84 // in the file system to be used for importing.
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
85 Create(kind JobKind, data string) (Job, error)
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
86 // Depends returns a list of ressources locked by this type of import.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
87 // Imports are run concurrently if they have disjoint sets
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
88 // of dependencies.
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
89 Depends() []string
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
90 // StageDone is called if an import is positively reviewed
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
91 // (state = accepted). This can be used to finalize the imported
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
92 // data to move it e.g from the staging area.
1328
d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1327
diff changeset
93 StageDone(context.Context, *sql.Tx, int64) error
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
94 }
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
96 idJob struct {
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
97 id int64
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
98 kind JobKind
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
99 user string
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
100 trysLeft sql.NullInt64
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
101 sendEmail bool
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
102 autoAccept bool
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
103 data string
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
104 }
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
105 )
987
3841509f6e9e Store job id alongside to job in job queue.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 979
diff changeset
106
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
107 const pollDuration = time.Second * 10
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
108
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
109 type importQueue struct {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
110 signalChan chan struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
111 creatorsMu sync.Mutex
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
112 creators map[JobKind]JobCreator
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
113 usedDeps map[string]struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
114 }
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
115
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
116 var iqueue = importQueue{
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
117 signalChan: make(chan struct{}),
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
118 creators: map[JobKind]JobCreator{},
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
119 usedDeps: map[string]struct{}{},
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
120 }
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
121
1189
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
122 var (
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
123 // ImportStateNames is a list of the states a job can be in.
1189
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
124 ImportStateNames = []string{
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
125 "queued",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
126 "running",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
127 "failed",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
128 "pending",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
129 "accepted",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
130 "declined",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
131 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
132 )
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1168
diff changeset
133
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
134 const (
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
135 queueUser = "sys_admin"
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
136
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
137 reEnqueueRunningSQL = `
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
138 UPDATE waterway.imports SET state = 'queued'::waterway.import_state
1409
74b0df86b6e6 Backend: Removed some trailing white space in some SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1392
diff changeset
139 WHERE state = 'running'::waterway.import_state`
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
140
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
141 insertJobSQL = `
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
142 INSERT INTO waterway.imports (
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
143 kind,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
144 due,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
145 trys_left,
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
146 username,
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
147 send_email,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
148 auto_accept,
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
149 data
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
150 ) VALUES (
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
151 $1,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
152 COALESCE($2, CURRENT_TIMESTAMP),
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
153 $3,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
154 $4,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
155 $5,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
156 $6,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
157 $7
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
158 ) RETURNING id`
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
159
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
160 selectJobSQL = `
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
161 SELECT
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
162 id,
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
163 kind,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
164 trys_left,
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
165 username,
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
166 send_email,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
167 auto_accept,
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
168 data
1409
74b0df86b6e6 Backend: Removed some trailing white space in some SQL statements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1392
diff changeset
169 FROM waterway.imports
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
170 WHERE
1718
8ddbedf296d7 Re-scheduled imports: be a bit more tolerant about start time.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1708
diff changeset
171 due <= CURRENT_TIMESTAMP + interval '5 seconds' AND
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
172 state = 'queued'::waterway.import_state AND enqueued IN (
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
173 SELECT min(enqueued)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
174 FROM waterway.imports
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
175 WHERE state = 'queued'::waterway.import_state AND
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
176 kind = ANY($1))
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
177 LIMIT 1`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
178
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
179 updateStateSQL = `
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
180 UPDATE waterway.imports SET state = $1::waterway.import_state
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
181 WHERE id = $2`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
182
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
183 updateStateSummarySQL = `
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
184 UPDATE waterway.imports SET
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
185 state = $1::waterway.import_state,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
186 summary = $2
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
187 WHERE id = $3`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
188
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
189 logMessageSQL = `
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
190 INSERT INTO waterway.import_logs (
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
191 import_id,
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
192 kind,
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
193 msg
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
194 ) VALUES (
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
195 $1,
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
196 $2::waterway.log_type,
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
197 $3
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
198 )`
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
199 )
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
200
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
201 func init() {
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
202 go iqueue.importLoop()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
203 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
204
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
205 // Error makes RetryError an error.
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
206 func (re *RetryError) Error() string {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
207 return re.Message
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
208 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
209
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
210 func (q *importQueue) registerJobCreator(kind JobKind, jc JobCreator) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
211 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
212 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
213 q.creators[kind] = jc
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 }
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
216 // FindJobCreator looks up a JobCreator in the global import queue.
1193
58acc343b1b6 Implemented the db stuff of the review process. Needs testing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1190
diff changeset
217 func FindJobCreator(kind JobKind) JobCreator {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1190
diff changeset
218 return iqueue.jobCreator(kind)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1190
diff changeset
219 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1190
diff changeset
220
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
221 // ImportKindNames is a list of the names of the imports the
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
222 // global import queue supports.
1495
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
223 func ImportKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
224 return iqueue.importKindNames()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
225 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
226
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1657
diff changeset
227 // HasImportKindName checks if the import queue supports a given kind.
1627
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
228 func HasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
229 return iqueue.hasImportKindName(kind)
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
230 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
231
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
232 //
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
233 func (q *importQueue) hasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
234 q.creatorsMu.Lock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
235 defer q.creatorsMu.Unlock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
236 return q.creators[JobKind(kind)] != nil
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
237 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1497
diff changeset
238
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
239 // RegisterJobCreator adds a JobCreator to the global import queue.
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
240 // This a good candidate to be called in a init function for
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
241 // a particular JobCreator.
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
242 func RegisterJobCreator(kind JobKind, jc JobCreator) {
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
243 log.Printf("info: register import job creator for kind '%s'\n", kind)
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
244 iqueue.registerJobCreator(kind, jc)
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
245 }
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
246
1495
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
247 func (q *importQueue) importKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
248 q.creatorsMu.Lock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
249 defer q.creatorsMu.Unlock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
250 names := make([]string, len(q.creators))
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
251 var i int
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
252 for kind := range q.creators {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
253 names[i] = string(kind)
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
254 i++
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
255 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
256 // XXX: Consider using sort.Strings to make output deterministic.
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
257 return names
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
258 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1409
diff changeset
259
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
260 func (idj *idJob) trys() int {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
261 if !idj.trysLeft.Valid {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
262 return -1
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
263 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
264 return int(idj.trysLeft.Int64)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
265 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
266
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
267 func (q *importQueue) jobCreator(kind JobKind) JobCreator {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
268 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
269 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
270 return q.creators[kind]
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
271 }
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
272
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
273 func (q *importQueue) addJob(
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
274 kind JobKind,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
275 due time.Time,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
276 trysLeft int,
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
277 user string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
278 sendEmail, autoAccept bool,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
279 data string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
280 ) (int64, error) {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
281 ctx := context.Background()
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
282 var id int64
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
283 if due.IsZero() {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
284 due = time.Now()
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
285 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
286 var tl sql.NullInt64
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
287 if trysLeft >= 0 {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
288 tl = sql.NullInt64{Int64: int64(trysLeft), Valid: true}
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
289 }
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
290 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
291 return conn.QueryRowContext(
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
292 ctx,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
293 insertJobSQL,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
294 string(kind),
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
295 due,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
296 tl,
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
297 user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
298 sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
299 autoAccept,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
300 data).Scan(&id)
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
301 })
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
302 if err == nil {
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
303 select {
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
304 case q.signalChan <- struct{}{}:
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
305 default:
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
306 }
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
307 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
308 return id, err
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
309 }
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
310
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
311 // AddJob adds a job to the global import queue to be executed
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
312 // as soon as possible after due.
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
313 // This is gone in a separate Go routine
1497
b41ad15cc55f Backend: Documented the internal API of the global import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1495
diff changeset
314 // so this will not block.
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
315 func AddJob(
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
316 kind JobKind,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
317 due time.Time,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
318 trysLeft int,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
319 user string,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
320 sendEmail, autoAccept bool,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
321 data string,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
322 ) (int64, error) {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
323 return iqueue.addJob(kind, due, trysLeft, user, sendEmail, autoAccept, data)
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
324 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
325
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
326 type logFeedback int64
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
327
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
328 func (lf logFeedback) log(kind, format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
329 ctx := context.Background()
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
330 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
331 _, err := conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
332 ctx, logMessageSQL, int64(lf), kind, fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
333 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
334 })
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
335 if err != nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
336 log.Printf("logging failed: %v\n", err)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
337 }
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
338 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
339
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
340 func (lf logFeedback) Info(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
341 lf.log("info", format, args...)
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
342 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
343
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
344 func (lf logFeedback) Warn(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
345 lf.log("warn", format, args...)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
346 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
347
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
348 func (lf logFeedback) Error(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
349 lf.log("error", format, args...)
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
350 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
351
992
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
352 func survive(fn func() error) func() error {
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
353 return func() (err error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
354 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
355 if p := recover(); p != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
356 err = fmt.Errorf("%v: %s", p, string(debug.Stack()))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
357 }
992
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
358 }()
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
359 return fn()
992
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
360 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
361 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 991
diff changeset
362
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
363 func reEnqueueRunning() error {
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
364 ctx := context.Background()
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
365 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
366 _, err := conn.ExecContext(ctx, reEnqueueRunningSQL)
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
367 return err
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
368 })
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
369 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
370
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
371 func (q *importQueue) fetchJob() (*idJob, error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
372
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
373 var which []string
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
374
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
375 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
376 nextCreator:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
377 for kind, jc := range q.creators {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
378 for _, d := range jc.Depends() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
379 if _, found := q.usedDeps[d]; found {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
380 continue nextCreator
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
381 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
382 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
383 which = append(which, string(kind))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
384 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
385 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
386
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
387 if len(which) == 0 {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
388 return nil, sql.ErrNoRows
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
389 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
390
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
391 var kinds pgtype.TextArray
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
392 if err := kinds.Set(which); err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
393 return nil, err
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
394 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
395
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
396 var ji idJob
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
397 ctx := context.Background()
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
398 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
399 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
400 if err != nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
401 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
402 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
403 defer tx.Rollback()
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
404 if err = tx.QueryRowContext(ctx, selectJobSQL, &kinds).Scan(
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
405 &ji.id,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
406 &ji.kind,
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
407 &ji.trysLeft,
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
408 &ji.user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
409 &ji.sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
410 &ji.autoAccept,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
411 &ji.data,
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
412 ); err != nil {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
413 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
414 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
415 _, err = tx.ExecContext(ctx, updateStateSQL, "running", ji.id)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
416 if err == nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
417 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
418 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
419 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
420 })
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
421 switch {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
422 case err == sql.ErrNoRows:
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
423 return nil, nil
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
424 case err != nil:
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
425 return nil, err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
426 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
427 return &ji, nil
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
428 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
429
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
430 func updateStateSummary(
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
431 ctx context.Context,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
432 id int64,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
433 state string,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
434 summary interface{},
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
435 ) error {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
436 var s sql.NullString
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
437 if summary != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
438 var b strings.Builder
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
439 if err := json.NewEncoder(&b).Encode(summary); err != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
440 return err
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
441 }
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
442 s = sql.NullString{String: b.String(), Valid: true}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
443 }
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
444 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
445 _, err := conn.ExecContext(ctx, updateStateSummarySQL, state, s, id)
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
446 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
447 })
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
448 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
449
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
450 func errorAndFail(id int64, format string, args ...interface{}) error {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
451 ctx := context.Background()
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
452 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
453 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
454 if err != nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
455 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
456 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
457 defer tx.Rollback()
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
458 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
459 ctx, logMessageSQL, id, "error", fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
460 if err != nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
461 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
462 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
463 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
464 ctx, updateStateSQL, "failed", id)
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
465 if err == nil {
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
466 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
467 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
468 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
469 })
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
470 return err
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
471 }
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
472
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
473 func (q *importQueue) importLoop() {
1000
14425e35e3c2 Wait with start of import queue until configuration is fully loaded.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 998
diff changeset
474 config.WaitReady()
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
475 // re-enqueue the jobs that are in state running.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
476 // They where in progess when the server went down.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
477 if err := reEnqueueRunning(); err != nil {
1751
c3a6aaf926c3 Import queue: Harmonized logging output.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1740
diff changeset
478 log.Printf("error: re-enqueuing failed: %v", err)
1005
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
479 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1003
diff changeset
480
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
481 for {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
482 var idj *idJob
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
483 var err error
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
484
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
485 for {
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
486 if idj, err = q.fetchJob(); err != nil && err != sql.ErrNoRows {
1751
c3a6aaf926c3 Import queue: Harmonized logging output.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1740
diff changeset
487 log.Printf("error: db: %v\n", err)
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
488 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
489 if idj != nil {
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
490 break
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
491 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
492 select {
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
493 case <-q.signalChan:
1003
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1000
diff changeset
494 case <-time.After(pollDuration):
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
495 }
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
496 }
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
497
1751
c3a6aaf926c3 Import queue: Harmonized logging output.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1740
diff changeset
498 log.Printf("info: starting import #%d\n", idj.id)
987
3841509f6e9e Store job id alongside to job in job queue.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 979
diff changeset
499
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
500 jc := q.jobCreator(idj.kind)
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
501 if jc == nil {
1010
8f23ec811afb Fixed and harmonized wording in importer queue a bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1005
diff changeset
502 errorAndFail(idj.id, "no creator for kind '%s' found", idj.kind)
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
503 continue
988
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
504 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 987
diff changeset
505
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
506 // Lock dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
507 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
508 for _, d := range jc.Depends() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
509 q.usedDeps[d] = struct{}{}
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
510 }
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
511 q.creatorsMu.Unlock()
991
a301d240905f Decoupled import job creation and job execution with a factory function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 989
diff changeset
512
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
513 go func(jc JobCreator, idj *idJob) {
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
514
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
515 // Unlock the dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
516 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
517 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
518 for _, d := range jc.Depends() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
519 delete(q.usedDeps, d)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
520 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
521 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
522 select {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
523 case q.signalChan <- struct{}{}:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
524 default:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
525 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
526 }()
1010
8f23ec811afb Fixed and harmonized wording in importer queue a bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1005
diff changeset
527
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
528 job, err := jc.Create(idj.kind, idj.data)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
529 if err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
530 errorAndFail(idj.id, "failed to create job for import #%d: %v",
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
531 idj.id, err)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
532 return
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
533 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
534
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
535 feedback := logFeedback(idj.id)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
536
1138
443fc80a315f Don't issue new lines at end of log messages when importing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1136
diff changeset
537 feedback.Info("import #%d started", idj.id)
998
75e65599ea52 Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 992
diff changeset
538
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
539 ctx := context.Background()
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
540 var summary interface{}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
541
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
542 errDo := survive(func() error {
1327
cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1193
diff changeset
543 return auth.RunAs(ctx, idj.user,
1168
930fdd8b474f Track successfull imports in a separate table to be able to remove them later.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1138
diff changeset
544 func(conn *sql.Conn) error {
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
545 var err error
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
546 summary, err = job.Do(ctx, idj.id, conn, feedback)
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
547 return err
1168
930fdd8b474f Track successfull imports in a separate table to be able to remove them later.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1138
diff changeset
548 })
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
549 })()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
550 if errDo != nil {
1138
443fc80a315f Don't issue new lines at end of log messages when importing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1136
diff changeset
551 feedback.Error("error do: %v", errDo)
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
552 }
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
553 // Should we try again?
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
554 retry, shouldRetry := errDo.(*RetryError)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
555
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
556 if shouldRetry && idj.trysLeft.Valid { // NULL -> limit less
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
557 if idj.trysLeft.Int64--; idj.trysLeft.Int64 <= 0 {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
558 shouldRetry = false
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
559 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
560 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
561
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
562 var errCleanup error
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
563 if !shouldRetry { // cleanup debris
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
564 errCleanup = survive(job.CleanUp)()
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
565 if errCleanup != nil {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
566 feedback.Error("error cleanup: %v", errCleanup)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
567 }
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
568 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
569
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
570 var state string
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
571 switch {
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
572 case errDo != nil || errCleanup != nil:
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
573 state = "failed"
1642
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
574 case idj.autoAccept:
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
575 state = "accepted"
49c04bb64e0a Import queue: Implemented auto-accept and email sending.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1627
diff changeset
576 default:
1190
e3de65179889 The imort queue has now six states:
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1189
diff changeset
577 state = "pending"
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
578 }
1392
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1328
diff changeset
579 if err := updateStateSummary(ctx, idj.id, state, summary); err != nil {
1751
c3a6aaf926c3 Import queue: Harmonized logging output.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1740
diff changeset
580 log.Printf("error: setting state of job %d failed: %v\n", idj.id, err)
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
581 }
1751
c3a6aaf926c3 Import queue: Harmonized logging output.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1740
diff changeset
582 log.Printf("info: import #%d finished: %s\n", idj.id, state)
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1642
diff changeset
583 if idj.sendEmail {
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1642
diff changeset
584 go sendNotificationMail(idj.user, jc.Description(), state, idj.id)
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1642
diff changeset
585 }
1708
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
586
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
587 if shouldRetry {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
588 nid, err := q.addJob(
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
589 idj.kind,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
590 retry.When, idj.trys(),
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
591 idj.user, idj.sendEmail, idj.autoAccept,
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
592 idj.data)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
593 if err != nil {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
594 log.Printf("error: retry enqueue failed: %v\n", err)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
595 } else {
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
596 log.Printf("info: re-enqueued job with id %d\n", nid)
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
597 }
49e047c2106e Imports: Made imports re-runnable if they fail.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
598 }
1136
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1017
diff changeset
599 }(jc, idj)
958
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
600 }
2818ad6c7d32 Started with import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
601 }