Mercurial > gemma

annotate cmd/octreediff/main.go @ 2472:db0e4ab57977 octree-diff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Turn new point cloud into WKB form of MultiPointZ.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Tue, 26 Feb 2019 12:22:04 +0100
parents 5bd3236f2b1f
children 19beb7d17337
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package main
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
17 "bytes"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "context"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "database/sql"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
20 "encoding/binary"
2468
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
21 "errors"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "flag"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "fmt"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "log"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
25 "math"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "runtime"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
27 "sync"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28 "time"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "gemma.intevation.de/gemma/pkg/common"
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 "gemma.intevation.de/gemma/pkg/octree"
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
32 "gemma.intevation.de/gemma/pkg/wkb"
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35 var (
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 bottleneck = flag.String("bottleneck", "", "name of the bottleneck")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37 first = flag.String("first", "", "date of the first sounding result")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38 second = flag.String("second", "", "date of the second sounding result")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41 const (
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42 loadOctreeSQL = `
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 SELECT sr.octree_index
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 FROM waterway.sounding_results sr JOIN waterway.bottlenecks bn
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 ON sr.bottleneck_id = bn.id
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 WHERE bn.bottleneck_id = $1 AND sr.date_info = $2::date
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 AND sr.octree_index IS NOT NULL`
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48 )
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 func check(err error) {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 if err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 log.Fatalf("error: %v\n", err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
56 type point struct {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
57 x float64
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
58 y float64
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
59 }
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
60
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
61 type pointMap map[point]float64
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
62
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
63 func (pm pointMap) asWKB() []byte {
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
64 size := 1 + 4 + 4 + len(pm)*(1+4+3*8)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
65
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
66 buf := bytes.NewBuffer(make([]byte, 0, size))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
67
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
68 binary.Write(buf, binary.LittleEndian, wkb.NDR)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
69 binary.Write(buf, binary.LittleEndian, wkb.MultiPointZ)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
70 binary.Write(buf, binary.LittleEndian, uint32(len(pm)))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
71
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
72 perPoint := bytes.NewBuffer(make([]byte, 0, 1+4))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
73 binary.Write(perPoint, binary.LittleEndian, wkb.NDR)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
74 binary.Write(perPoint, binary.LittleEndian, wkb.PointZ)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
75 hdr := perPoint.Bytes()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
76
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
77 for p, z := range pm {
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
78 buf.Write(hdr)
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
79 binary.Write(buf, binary.LittleEndian, math.Float64bits(p.x))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
80 binary.Write(buf, binary.LittleEndian, math.Float64bits(p.y))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
81 binary.Write(buf, binary.LittleEndian, math.Float64bits(z))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
82 }
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
83
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
84 return buf.Bytes()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
85 }
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
86
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
87 func sliceWork(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
88 vs []octree.Vertex,
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
89 dst pointMap,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
90 fn func([]octree.Vertex, func([]octree.Vertex) []octree.Vertex),
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
91 ) {
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
92 n := runtime.NumCPU()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
93
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
94 wg := new(sync.WaitGroup)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
95
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
96 slices := make(chan []octree.Vertex)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
97 out := make(chan []octree.Vertex)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
98
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
99 pool := make(chan []octree.Vertex, n)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
100
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
101 const pageSize = 2048
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
102
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
103 turn := func(p []octree.Vertex) []octree.Vertex {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
104 if p != nil {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
105 out <- p
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
106 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
107 select {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
108 case p = <-pool:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
109 default:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
110 p = make([]octree.Vertex, 0, pageSize)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
111 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
112 return p
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
113 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
114
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
115 for i := 0; i < n; i++ {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
116 wg.Add(1)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
117 go func() {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
118 defer wg.Done()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
119 for slice := range slices {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
120 fn(slice, turn)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
121 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
122 }()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
123 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
124 done := make(chan struct{})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
125 go func() {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
126 defer close(done)
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
127 for s := range out {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
128 for i := range s {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
129 v := &s[i]
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
130 key := point{x: v.X, y: v.Y}
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
131 if z, found := dst[key]; found {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
132 dst[key] = (z + v.Z) * 0.5
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
133 } else {
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
134 dst[key] = v.Z
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
135 }
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
136 }
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
137 select {
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
138 case pool <- s[:0:pageSize]:
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
139 default:
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
140 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
141 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
142 }()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
143
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
144 size := len(vs)/n + 1
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
145 for len(vs) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
146 var l int
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
147 if len(vs) < size {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
148 l = len(vs)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
149 } else {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
150 l = size
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
151 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
152 slices <- vs[:l]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
153 vs = vs[l:]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
154 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
155 close(slices)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
156 wg.Wait()
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
157 close(out)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
158 <-done
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
159 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
160
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
161 func process(bottleneck string, firstDate, secondDate time.Time) error {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162 start := time.Now()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 defer func() { log.Printf("processing took %v\n", time.Since(start)) }()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 ctx := context.Background()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 return run(func(db *sql.DB) error {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 conn, err := db.Conn(ctx)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 if err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
171 return err
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173 defer conn.Close()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175 type load struct {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 date time.Time
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
177 data []byte
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
178 err *error
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179 dst **octree.Tree
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
180 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
181
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
182 out := make(chan *load)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
183 wg := new(sync.WaitGroup)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
184
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
185 n := runtime.NumCPU()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
186 if n > 2 {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
187 n = 2
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
188 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
189
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
190 for i := 0; i < n; i++ {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
191 wg.Add(1)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
192 go func() {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
193 defer wg.Done()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
194 for l := range out {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
195 if *l.err == nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
196 *l.dst, *l.err = octree.Deserialize(l.data)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
197 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
198 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
199 }()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
200 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
201
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
202 var firstErr, secondErr error
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
203 var first, second *octree.Tree
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
204
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205 for _, l := range []*load{
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 {date: firstDate, dst: &first, err: &firstErr},
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 {date: secondDate, dst: &second, err: &secondErr},
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 } {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 var data []byte
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 if err := conn.QueryRowContext(
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 ctx,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 loadOctreeSQL,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213 bottleneck,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 l.date,
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 ).Scan(&data); err != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 *l.err = err
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217 } else {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
218 l.data = data
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
220 out <- l
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222 close(out)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
224 wg.Wait()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
225
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
226 if firstErr != nil || secondErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 if firstErr != nil && secondErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228 return fmt.Errorf("%v, %v", firstErr, secondErr)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
229 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
230 if firstErr != nil {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
231 return firstErr
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
233 return secondErr
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
234 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
235
2468
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
236 if first.EPSG != second.EPSG {
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
237 return errors.New("EPSG codes mismatch. Needs transformation slow pass.")
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
238 }
27cc04bbda46 Bail out if the EPSG codes of the datasets don't match.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2467
diff changeset
239
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
240 now := time.Now()
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
241 log.Printf("loading took %v\n", now.Sub(start))
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
242 last := now
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
243
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
244 firstVs, secondVs := first.Vertices(), second.Vertices()
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
245
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
246 result := make(pointMap, len(firstVs)+len(secondVs))
2466
a1e751c08c56 Calculate difference on single core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2465
diff changeset
247
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
248 sliceWork(
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
249 firstVs,
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
250 result,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
251 func(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
252 slice []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
253 turn func([]octree.Vertex) []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
254 ) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
255 p := turn(nil)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
256 for i := range slice {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
257 v := &slice[i]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
258 if z, found := second.Value(v.X, v.Y); found {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
259 p = append(p, octree.Vertex{v.X, v.Y, v.Z - z})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
260 if len(p) == cap(p) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
261 p = turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
262 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
263 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
264 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
265 if len(p) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
266 turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
267 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
268 })
2466
a1e751c08c56 Calculate difference on single core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2465
diff changeset
269
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
270 sliceWork(
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
271 secondVs,
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
272 result,
2469
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
273 func(
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
274 slice []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
275 turn func([]octree.Vertex) []octree.Vertex,
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
276 ) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
277 p := turn(nil)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
278 for i := range slice {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
279 v := &slice[i]
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
280 if z, found := first.Value(v.X, v.Y); found {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
281 p = append(p, octree.Vertex{v.X, v.Y, z - v.Z})
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
282 if len(p) == cap(p) {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
283 p = turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
284 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
285 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
286 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
287 if len(p) > 0 {
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
288 turn(p)
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
289 }
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
290 })
1312953defab Do difference calculation multi-core.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2468
diff changeset
291
2470
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
292 now = time.Now()
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
293 log.Printf("setting in took %v\n", now.Sub(last))
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
294 last = now
5bd3236f2b1f Integrate deduplication with setting in. Even if there is more contention in total it seems to be cheaper.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2469
diff changeset
295 log.Printf("num points: %d\n", len(result))
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
296
2472
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
297 data := result.asWKB()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
298
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
299 now = time.Now()
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
300 log.Printf("turing into WKB took %v\n", now.Sub(last))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
301 last = now
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
302
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
303 log.Printf("WKB size %.3fMB\n", float64(len(data))/(1024*1024))
db0e4ab57977 Turn new point cloud into WKB form of MultiPointZ.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2470
diff changeset
304
2465
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
305 return nil
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
306 })
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
307 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
308
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
309 func main() {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
310
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
311 flag.Parse()
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
312
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
313 firstDate, err := time.Parse(common.DateFormat, *first)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
314 check(err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
315 secondDate, err := time.Parse(common.DateFormat, *second)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
316 check(err)
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
317
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
318 if *bottleneck == "" {
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
319 log.Fatalln("Missing bottleneck name")
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
320 }
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
321
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
322 check(process(*bottleneck, firstDate, secondDate))
86c7a023400e Started experimental octree diff branch.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
323 }