Possible Bug in Multiply and Scale

[dmbrubac]

Daniel;
I'm working through various scenarios to figure out exactly what is going on and I will report more, but I wanted to draw your attention early to something that looks like a serious bug.

When opening a LAZ cloud, we have an opportunity to set offsets. I work in UTM so my offsets in XY are pretty big though I always set them to even 500m or 1000m values. In Z I'm usually in the 100m to 350m range, so I prefer to set the offset to 0.

Next I use the SOR tool to remove noise. I don't think this has anything to do with the problem but I'm including it for completeness. My data comes from a UAV via Pix4D, so there is noise there for sure.

The next step is new in my process and since the problem is also new, I strongly suspect it is related. I apply a Z only scale of 5 and I uncheck 'keep in place'. What I want is a 5x exaggeration in Z, scaled from 0. looking at the point values and the documentation suggests this is exactly what happens - all good so far.

Next I manually remove the man-made bits from my cloud. I'm looking for a nice clean DTM and automated processes only go so far. I segment and then use a combo of limiting Coord.Z and filtering with tilting the cloud and manually segmenting out bits I don't want. This process gets repeated over and over until everything is clean.

I regularly merge the cleaned (or trimmed) bits of the original cloud and save the cloud as ABC Trimmed.LAZ and the remaining cloud as ABC Remaining.LAZ. Occasionally I must close down CC or even stop working on the project for hours or days. When I start again I open the two clouds, export Z to a scalar and continue.

After everything is done, I scale the Z by 0.2 with 'keep in place' unchecked.

Here is where the problem comes in. Since I have ground targets with precise heights, I should be able to confirm that the target locations have points around them that are close (say +/- 5 cm) to my GPS elevations. If I don't do the scaling I can definitely confirm my elevations are still good. When I use scaling, however, the elevations can be quite far off (+/- 1 metre is what I've seen). Not only that, different control points will be off by different amounts, so I can't just shift up or down.

Furthermore, if, instead of a single ABC Trimmed.laz I have multiple (e.g. ABC Trimmed East.laz and ABC Trimmed West.laz), there is an elevation shift between the two pieces when I'm done.

It's almost as if each piece I work on separately gets scaled a different amount despite the fact they've all been merged back together.

As I said, I'm not completely sure what is happening and will keep investigating, but I wanted to let you know about a potentially serious issue.

Also, can you add additional numbers to the beta versions? I downloaded the beta again a couple weeks ago, but I can't tell you anything beyond 2.6.3 beta as shown in Help / About (I'd like to see a build number after the 2.6.3 if possible). The newest entry in history.txt is v2.7.beta ??/??/201? - Polyline tracing tool

[daniel]

Hi,

For the beta version, you can simply give me the date of the executable (CloudCompare.exe). It's sufficient for me to recognize the version.

When you say "I work in UTM so my offsets in XY are pretty big though I always set them to even 500m or 1000m values" I assume that the shifted coordinates are in this [500 ; 1000 m.] range? (I bet it is but I prefer to be sure as otherwise it would definitely cause precision issues).

Otherwise the other potential precision issue is with the way LAZ files are compressed. What LAZ 'scale' are you using when you save the files? (I believe you should see it in the dialog that appear at saving time).

I explain: LAZ files are compressed by dividing the coordinates by the 'scale' (e.g. 0.1 or 0.01) and then converting the floating point values to integer values. It is basically equivalent to ignoring all digits smaller than the 'scale'. Therefore it could be problematic if you multiply the coordinates yourself, save a LAZ file with a given precision (which means that you may lose some information) and eventually reload the cloud and apply the inverse scaling. At the end you would miss a small quantity multiplied by 5 each times.

The issue is that it shouldn't cause such big errors (apart if you use a huge scale of course). Anyway it is a potential source of errors.

Another problem that may be linked to all this is that if you apply a scaling to a shifted cloud, then the shift vector is also scaled. But in your case if the shift component along Z is 0 then it shouldn't matter...

If you can isolate the issue on a single cloud and with a simple process (e.g. load cloud, scale it, save it as LAZ, reopen, multiply back and compare it to the original cloud) then I would be very interested in getting the original cloud (I'll then be able to check if the issue is with the LAZ compression).

[dmbrubac]

Thanks for the info Daniel.

Regarding the date of the executable, I installed the most recent version yesterday but I thought the date on the version I used when encountering this issue was Jan 29 or 30. Does that make sense? The date on the one I have now is Feb 13.

Regarding the offsets, a picture is worth 1000 words. This is a typical setting for me.

Shift and Scale settings

Shift and Scale.JPG (48.14 KiB) Viewed 3859 times

EDIT: Oops! I was too quick giving you a sample. The Y value shouldn't be 8839.758 but just 839.758. I don't remove the 1000's place.

I usually set the file precision to 0.01m and do a 'remove duplicates' to the same amount before applying the scale. With these settings the data should round-trim without a problem, though by the sounds of it I should be saving and loading from you BIN format during interim steps, correct?

I will endeavor to investigate more.

[daniel]

The version shouldn't matter too much as in this case none of the involved components have been modified for long.

And indeed the Global shift parameters are ok.

Otherwise it's clear that using the BIN format would be safer. But I would be very interested to understand what's going on even if it comes from the LAZ format.