Mercurial > gemma

annotate cmd/octreediff/main.go @ 2476:efe332e985b9 octree-diff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Calculate the iso lines of the difference.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Thu, 28 Feb 2019 11:51:31 +0100
parents 19beb7d17337
children c85b16db8a02
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package main
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
17 "bytes"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "context"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "database/sql"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
20 "encoding/binary"
2468
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
21 "errors"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "flag"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "fmt"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "log"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
25 "math"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "runtime"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
27 "sync"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28 "time"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "gemma.intevation.de/gemma/pkg/common"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 "gemma.intevation.de/gemma/pkg/octree"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
32 "gemma.intevation.de/gemma/pkg/wkb"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35 var (
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 bottleneck = flag.String("bottleneck", "", "name of the bottleneck")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37 first = flag.String("first", "", "date of the first sounding result")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38 second = flag.String("second", "", "date of the second sounding result")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41 const (
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42 loadOctreeSQL = `
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 SELECT sr.octree_index
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 FROM waterway.sounding_results sr JOIN waterway.bottlenecks bn
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 ON sr.bottleneck_id = bn.id
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 WHERE bn.bottleneck_id = $1 AND sr.date_info = $2::date
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 AND sr.octree_index IS NOT NULL`
2473
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
48
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
49 triangulateSQL = `
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
50 SELECT ST_AsBinary(ST_DelaunayTriangles(ST_GeomFromWKB($1, $2::int), 0, 2))
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
51 `
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 func check(err error) {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 if err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 log.Fatalf("error: %v\n", err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
60 type point struct {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
61 x float64
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
62 y float64
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
63 }
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
64
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
65 type pointMap map[point]float64
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
66
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
67 func (pm pointMap) asWKB() []byte {
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
68 size := 1 + 4 + 4 + len(pm)*(1+4+3*8)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
69
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
70 buf := bytes.NewBuffer(make([]byte, 0, size))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
71
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
72 binary.Write(buf, binary.LittleEndian, wkb.NDR)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
73 binary.Write(buf, binary.LittleEndian, wkb.MultiPointZ)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
74 binary.Write(buf, binary.LittleEndian, uint32(len(pm)))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
75
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
76 perPoint := bytes.NewBuffer(make([]byte, 0, 1+4))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
77 binary.Write(perPoint, binary.LittleEndian, wkb.NDR)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
78 binary.Write(perPoint, binary.LittleEndian, wkb.PointZ)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
79 hdr := perPoint.Bytes()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
80
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
81 for p, z := range pm {
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
82 buf.Write(hdr)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
83 binary.Write(buf, binary.LittleEndian, math.Float64bits(p.x))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
84 binary.Write(buf, binary.LittleEndian, math.Float64bits(p.y))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
85 binary.Write(buf, binary.LittleEndian, math.Float64bits(z))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
86 }
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
87
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
88 return buf.Bytes()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
89 }
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
90
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
91 func sliceWork(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
92 vs []octree.Vertex,
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
93 dst pointMap,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
94 fn func([]octree.Vertex, func([]octree.Vertex) []octree.Vertex),
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
95 ) {
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
96 n := runtime.NumCPU()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
97
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
98 wg := new(sync.WaitGroup)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
99
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
100 slices := make(chan []octree.Vertex)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
101 out := make(chan []octree.Vertex)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
102
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
103 pool := make(chan []octree.Vertex, n)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
104
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
105 const pageSize = 2048
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
106
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
107 turn := func(p []octree.Vertex) []octree.Vertex {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
108 if p != nil {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
109 out <- p
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
110 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
111 select {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
112 case p = <-pool:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
113 default:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
114 p = make([]octree.Vertex, 0, pageSize)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
115 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
116 return p
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
117 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
118
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
119 for i := 0; i < n; i++ {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
120 wg.Add(1)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
121 go func() {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
122 defer wg.Done()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
123 for slice := range slices {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
124 fn(slice, turn)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
125 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
126 }()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
127 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
128 done := make(chan struct{})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
129 go func() {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
130 defer close(done)
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
131 for s := range out {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
132 for i := range s {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
133 v := &s[i]
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
134 key := point{x: v.X, y: v.Y}
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
135 if z, found := dst[key]; found {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
136 dst[key] = (z + v.Z) * 0.5
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
137 } else {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
138 dst[key] = v.Z
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
139 }
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
140 }
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
141 select {
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
142 case pool <- s[:0:pageSize]:
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
143 default:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
144 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
145 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
146 }()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
147
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
148 size := len(vs)/n + 1
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
149 for len(vs) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
150 var l int
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
151 if len(vs) < size {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
152 l = len(vs)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
153 } else {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
154 l = size
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
155 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
156 slices <- vs[:l]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
157 vs = vs[l:]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
158 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
159 close(slices)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
160 wg.Wait()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
161 close(out)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
162 <-done
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
163 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
164
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 func process(bottleneck string, firstDate, secondDate time.Time) error {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 start := time.Now()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 defer func() { log.Printf("processing took %v\n", time.Since(start)) }()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 ctx := context.Background()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
171 return run(func(db *sql.DB) error {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173 conn, err := db.Conn(ctx)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 if err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175 return err
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
177 defer conn.Close()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
178
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179 type load struct {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
180 date time.Time
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
181 data []byte
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
182 err *error
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
183 dst **octree.Tree
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
184 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
185
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
186 out := make(chan *load)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
187 wg := new(sync.WaitGroup)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
188
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
189 n := runtime.NumCPU()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
190 if n > 2 {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
191 n = 2
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
192 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
193
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
194 for i := 0; i < n; i++ {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
195 wg.Add(1)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
196 go func() {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
197 defer wg.Done()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
198 for l := range out {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
199 if *l.err == nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
200 *l.dst, *l.err = octree.Deserialize(l.data)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
201 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
202 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
203 }()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
204 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 var firstErr, secondErr error
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 var first, second *octree.Tree
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 for _, l := range []*load{
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 {date: firstDate, dst: &first, err: &firstErr},
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 {date: secondDate, dst: &second, err: &secondErr},
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 } {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213 var data []byte
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 if err := conn.QueryRowContext(
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 ctx,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 loadOctreeSQL,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217 bottleneck,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
218 l.date,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 ).Scan(&data); err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
220 *l.err = err
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 } else {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222 l.data = data
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
224 out <- l
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
225 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
226 close(out)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228 wg.Wait()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
229
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
230 if firstErr != nil || secondErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
231 if firstErr != nil && secondErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232 return fmt.Errorf("%v, %v", firstErr, secondErr)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
233 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
234 if firstErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
235 return firstErr
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
236 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
237 return secondErr
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
238 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
239
2468
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
240 if first.EPSG != second.EPSG {
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
241 return errors.New("EPSG codes mismatch. Needs transformation slow pass.")
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
242 }
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
243
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
244 now := time.Now()
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
245 log.Printf("loading took %v\n", now.Sub(start))
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
246 last := now
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
247
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
248 firstVs, secondVs := first.Vertices(), second.Vertices()
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
249
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
250 result := make(pointMap, len(firstVs)+len(secondVs))
2466
a1e751c08c56 Calculate difference on single core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2465
diff changeset
251
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
252 sliceWork(
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
253 firstVs,
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
254 result,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
255 func(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
256 slice []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
257 turn func([]octree.Vertex) []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
258 ) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
259 p := turn(nil)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
260 for i := range slice {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
261 v := &slice[i]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
262 if z, found := second.Value(v.X, v.Y); found {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
263 p = append(p, octree.Vertex{v.X, v.Y, v.Z - z})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
264 if len(p) == cap(p) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
265 p = turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
266 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
267 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
268 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
269 if len(p) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
270 turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
271 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
272 })
2466
a1e751c08c56 Calculate difference on single core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2465
diff changeset
273
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
274 sliceWork(
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
275 secondVs,
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
276 result,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
277 func(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
278 slice []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
279 turn func([]octree.Vertex) []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
280 ) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
281 p := turn(nil)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
282 for i := range slice {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
283 v := &slice[i]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
284 if z, found := first.Value(v.X, v.Y); found {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
285 p = append(p, octree.Vertex{v.X, v.Y, z - v.Z})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
286 if len(p) == cap(p) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
287 p = turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
288 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
289 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
290 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
291 if len(p) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
292 turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
293 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
294 })
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
295
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
296 now = time.Now()
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
297 log.Printf("setting in took %v\n", now.Sub(last))
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
298 last = now
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
299 log.Printf("num points: %d\n", len(result))
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
300
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
301 data := result.asWKB()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
302
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
303 now = time.Now()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
304 log.Printf("turing into WKB took %v\n", now.Sub(last))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
305 last = now
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
306
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
307 log.Printf("WKB size %.3fMB\n", float64(len(data))/(1024*1024))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
308
2473
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
309 var tin octree.Tin
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
310
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
311 if err := conn.QueryRowContext(
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
312 ctx,
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
313 triangulateSQL,
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
314 data,
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
315 first.EPSG,
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
316 ).Scan(&tin); err != nil {
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
317 return err
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
318 }
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
319
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
320 now = time.Now()
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
321 log.Printf("triangulation took %v\n", now.Sub(last))
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
322 last = now
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
323
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
324 builder := octree.NewBuilder(&tin)
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
325 builder.Build()
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
326
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
327 now = time.Now()
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
328 log.Printf("building octree took %v\n", now.Sub(last))
19beb7d17337 Triangulate and build octree index.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2472
diff changeset
329 last = now
2476
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
330
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
331 tree := builder.Tree()
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
332
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
333 log.Printf("min/max: %f %f\n", tree.Min.Z, tree.Max.Z)
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
334
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
335 var heights []float64
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
336
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
337 switch {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
338 case tree.Min.Z >= 0: // All values positive.
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
339 for v := 0.0; v <= tree.Max.Z; v += 0.1 {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
340 if v >= tree.Min.Z {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
341 heights = append(heights, v)
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
342 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
343 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
344 case tree.Max.Z <= 0: // All values negative.
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
345 for v := 0.0; v >= tree.Min.Z; v -= 0.1 {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
346 if v <= tree.Max.Z {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
347 heights = append(heights, v)
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
348 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
349 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
350 default: // Positive and negative.
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
351 for v := 0.1; v <= tree.Max.Z; v += 0.1 {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
352 heights = append(heights, v)
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
353 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
354 for i, j := 0, len(heights)-1; i < j; i, j = i+1, j-1 {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
355 heights[i], heights[j] = heights[j], heights[i]
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
356 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
357 for v := 0.0; v >= tree.Min.Z; v -= 0.1 {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
358 heights = append(heights, v)
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
359 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
360 }
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
361
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
362 var dataSize int
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
363
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
364 octree.DoContours(tree, heights, func(res *octree.ContourResult) {
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
365 // TODO: Store them.
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
366 log.Printf("%f: segments: %d\n", res.Height, len(res.Lines))
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
367 dataSize += len(res.Lines.AsWKB2D())
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
368 })
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
369
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
370 now = time.Now()
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
371 log.Printf("Number of iso lines: %d\n", len(heights))
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
372 log.Printf("Total WKB size: %.2fMB\n", float64(dataSize)/(1024*1024))
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
373 log.Printf("generating iso lines took %v\n", now.Sub(last))
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
374 last = now
efe332e985b9 Calculate the iso lines of the difference.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2473
diff changeset
375
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
376 return nil
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
377 })
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
378 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
379
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
380 func main() {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
381
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
382 flag.Parse()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
383
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
384 firstDate, err := time.Parse(common.DateFormat, *first)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
385 check(err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
386 secondDate, err := time.Parse(common.DateFormat, *second)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
387 check(err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
388
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
389 if *bottleneck == "" {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
390 log.Fatalln("Missing bottleneck name")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
391 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
392
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
393 check(process(*bottleneck, firstDate, secondDate))
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
394 }