Zerene : to raid or not to raid ?

Macro_Cosmos · Post by **Macro_Cosmos** » Sun Apr 05, 2020 2:46 am

Would RAID help with frame by frame retouching? It is too laggy on my laptop for me to be bothered. I'm running an i7-8750h, stacking is done from my pcie ssd, Evo970plus, should be 3.0. Read and write is fast but the retouching is sluggish.

I guess I'll have to build the PC to be able to decide. "laggy" and "sluggish" are very subjective.

Adalbert · Post by **Adalbert** » Sun Apr 05, 2020 9:16 am

Hi Macro_Cosmos,
Do you know how to move Zerene completely to the RAM ?
BR, ADi

rjlittlefield · Post by **rjlittlefield** » Sun Apr 05, 2020 10:21 am

Macro_Cosmos wrote:Would RAID help with frame by frame retouching?

At this time, every time you select a new file as source for retouching, Zerene Stacker goes through a long process of reading the original source image from disk and transforming it for alignment and brightness correction. Then if you're using the default Details brush, it also converts the image to Laplacian pyramid representation. All of this is computationally intensive. The only effect of RAID would be to reduce the original reading time, if you're using uncompressed TIFF source.

This choice of dataflow is rooted in history, when images were large compared to RAM and most disks were so slow that it was faster to recompute than to pay the I/O cost of caching the 12- or 16-bytes-per-pixel representation that is used by the actual retouching process.

In the modern era of fast storage and large RAM, it should be feasible to rework the dataflow so as to cache in RAM or quickly transfer to RAID or SSD. But that's not done yet.

As a matter of practice, at this time the best ways to make retouching tolerable are still the hints at https://zerenesystems.com/cms/stacker/d ... retouching . That and slabbing, if your source stacks are deep.

Adalbert wrote:Do you know how to move Zerene completely to the RAM ?

The only way I know would be to define a "RAM disk" and tell Zerene Stacker to create new projects on that device. However, the program would still be using dataflows similar to what's described above, so I would not expect big gains in performance.

--Rik

mjkzz · Post by **mjkzz** » Fri Apr 10, 2020 9:34 pm

my 2 cents, just be careful with RAID 0, if you lose one drive, you lose all, basically you increase the chance of losing all data, not a good idea as storage configuration or to archive images over a long run.

One way to speed up stacking process is to run multiple copies of Zerene [edit] just read Rik's post, working on slabs of same stack or multiple stacks. Zerene is VERY well behaved and you can strike a balance between CPU and data rate.

Adalbert · Post by **Adalbert** » Sat Apr 11, 2020 12:50 am

Disadvantages of RAID-0 are clear.
The idea with substacks in separate sessions is really good because I usually create substacks with PMax and then calculate the results with DMap.
BTW, my problem is not only Zerene but Lightroom too.

mjkzz · Post by **mjkzz** » Sat Apr 11, 2020 4:49 am

ah, the title says Zerene specifically . . .

for me, I tend not to buy state of art hardware, here is my solution editing 4K video, maybe it applies to your lightroom situation. I will create a separate post thread with this to avoid hijacking your thread

I bought an old (10+ years old) Dell R720 server, it comes with dual E5-2697 CPU (12 cores EACH, total 24 cores), 256GB RAM (will upgrade it to 512GB), 4 10Gb ethernet ports, added a fiber card with 2 10Gb ports because I could not find a good copper card for my PC. All for about 1000USD, then created RAM disk with 224GB capacity.

Over the fiber, I can get about 2GB (giga BYTE) max bandwidth (see blog I wrote a long while ago).

With this setup, pretty comfortable editing 4K video when I put those temp files on the RAM disk and over the fiber, even on an i5 machine.

of course, that old server is not just for video editing, for other computational tasks, too, particularly those that needs a lot of memory.

Macro_Cosmos · Post by **Macro_Cosmos** » Sat Apr 11, 2020 5:06 am

On my laptop, I have tried running several instances of ZS for shallow stacks intended for panorama usage. About 5-10 exposures each stack. What ends up happening is all of them crashing.

My laptop can only really handle 2 instances. I'm not greedy, there's enough time to pass by these days.

Adalbert · Post by **Adalbert** » Sat Apr 11, 2020 5:19 am

Not bad :-)

I have already ordered the Ryzen 9 3900x and some components.
https://www.photomacrography.net/forum/ ... highlight=
My mainboard allows only 128GB RAM but contains 2 PCIe 4. slots for M2.
Actually I wanted to build a RAID-0 from M2 NVMe-SSD but I’ll start with one:
“Corsair Force Series Gen.4 PCIe MP600 1TB”
and then maybe will use Asus Hyper M.2.

BTW, because of the LR and PS I use Windows 10
and have some problems with the temperature of the CPU because of Zerene :-)
But is no problem under Linux.

BR, ADi

mjkzz · Post by **mjkzz** » Sat Apr 11, 2020 5:36 am

yes, I had thermal issues before with an old i7 CPU running Win 10, and I did not know what was going on, until Rik told me to run a numeric app (sorry I forgot the name of it) . . . so I re-applied thermal paste on the CPU and that fixed the problem.

Macro_Cosmos · Post by **Macro_Cosmos** » Sat Apr 11, 2020 7:36 am

mjkzz wrote:yes, I had thermal issues before with an old i7 CPU running Win 10, and I did not know what was going on, until Rik told me to run a numeric app (sorry I forgot the name of it) . . . so I re-applied thermal paste on the CPU and that fixed the problem.

Numeric App sounds like CPU-Z, fits the name perfectly.

Chris S. · Post by **Chris S.** » Sat Apr 11, 2020 3:06 pm

Macro_Cosmos wrote:
mjkzz wrote:yes, I had thermal issues before with an old i7 CPU running Win 10, and I did not know what was going on, until Rik told me to run a numeric app (sorry I forgot the name of it) . . . so I re-applied thermal paste on the CPU and that fixed the problem.
Numeric App sounds like CPU-Z, fits the name perfectly.

Or Prime95 in "Stress test" mode, which is what I use.

--Chris S.

mjkzz · Post by **mjkzz** » Sat Apr 11, 2020 4:58 pm

Yeah, Prime95 is the one.

What happened was, just to share it, I had a cool looking CPU heat sink from another project, decided to put it on the i7, the mistake was I simply added some thermal paste over existing leftover (semi-dried up) . . . then it started to crash whenever I ran Zerene

After Prime95 advised by Rik, I took the CPU out, cleaned its surface thoroughly, applied very thin layer of thermal paste, it never crashed again.

Chris S. · Post by **Chris S.** » Sun Apr 12, 2020 2:14 am

mjkzz wrote:Yeah, Prime95 is the one.

What happened was, just to share it, I had a cool looking CPU heat sink from another project, decided to put it on the i7, the mistake was I simply added some thermal paste over existing leftover (semi-dried up) . . . then it started to crash whenever I ran Zerene

After Prime95 advised by Rik, I took the CPU out, cleaned its surface thoroughly, applied very thin layer of thermal paste, it never crashed again.

When PCs crash while running hard with stable software, the cause is almost always heat build-up. One of the most common causes of heat build-up is a problem with the thermal paste between the CPU and CPU cooler. The fix for this is thoroughly cleaning off any existing paste, finger prints, dust, or even a stray hair--getting CPU and heat sink utterly, scrupulously clean with (in my case) acetone and PEC pads. Then applying a very thin layer of new, quality thermal paste, and putting everything back together. Later, if the CPU-cooler contact needs to be broken for any reason, the old paste should be cleaned off thoroughly, and a new layer applied.

Macro_Cosmos wrote:On my laptop, I have tried running several instances of ZS for shallow stacks intended for panorama usage. About 5-10 exposures each stack. What ends up happening is all of them crashing.

My laptop can only really handle 2 instances. I'm not greedy, there's enough time to pass by these days.

Macro Cosmos,

Laptops have much less robust cooling systems than desktops or towers. With most laptops, cooling consists of a narrow air tunnel blowing across a heat sink that cools the CPU. Over time, a felt-like plug of dust often forms on the heat sink, dramatically reducing air flow. One symptom of this that you will hear the internal fan run more than before, especially when the computer is doing work. Another symptom is that the laptop will crash when running hard. Remember, crashes are almost always caused by overheating.

The remedy is not to try to blow the dust of out the laptop; by the time it has formed a felt-like wad, it can’t be dislodged by air. The solution is to open up the laptop and remove the mat of dust. Usually this also disturbs the heat sink paste, so here again, the old past should be cleaned off and quality heat paste thinly and carefully applied. When this is done, a laptop should be able to run very hard without crashing. The caveat is that accessing the heat sink runs from trivially easy on some laptops, to terribly difficult on others. I always look on YouTube for disassembly instructions before I attempt this repair on an unknown model.

--Chris S.

Macro_Cosmos · Post by **Macro_Cosmos** » Sun Apr 12, 2020 4:20 am

Chris S. wrote:
mjkzz wrote:Yeah, Prime95 is the one.
"Numerical", why did I somehow think it was "alphabetical"... ehh.

Chris S. wrote: Laptops have much less robust cooling systems than desktops or towers. With most laptops, cooling consists of a narrow air tunnel blowing across a heat sink that cools the CPU. Over time, a felt-like plug of dust often forms on the heat sink, dramatically reducing air flow. One symptom of this that you will hear the internal fan run more than before, especially when the computer is doing work. Another symptom is that the laptop will crash when running hard. Remember, crashes are almost always caused by overheating.

The remedy is not to try to blow the dust of out the laptop; by the time it has formed a felt-like wad, it can’t be dislodged by air. The solution is to open up the laptop and remove the mat of dust. Usually this also disturbs the heat sink paste, so here again, the old past should be cleaned off and quality heat paste thinly and carefully applied. When this is done, a laptop should be able to run very hard without crashing. The caveat is that accessing the heat sink runs from trivially easy on some laptops, to terribly difficult on others. I always look on YouTube for disassembly instructions before I attempt this repair on an unknown model.

--Chris S.
Yeah laptops in general don't have good cooling, even though mine is an Alienware that has a massive chassis and extremely heavy duty thick heatpipes. Mine is also elevated which makes ventilation better.

The accumulated dust does make an interesting microscopy subject , under XPL as well.

For thermal paste, I always recommend non-healing brand name ones. They do make a difference over time. A blop that's the size of a pea in the middle works. Heatsink paste residue can be cleaned off with rubbing alcohol or multi-purpose cleaners. OEM paste are rarely good. Another good application method is a thin "X" shape with small dots at the intersection, ie "※".

Liquid metal drops cpu temperatures by about 5deg C, hardly worth the hassle though.

If anyone wants to completely disassemble a laptop for cleaning and repaste, do follow online guides.

Alienware has a PDF that's extensive and step-by-step. Most manufactures don't want people opening the machine.

Never mix up screw lengths! Forcing a long screw into a shallow thread will result in a potential disaster.

Additionally, sometimes you need to reinstall the CMOS battery of the laptop (unplug, replug), or it won't boot. I don't know why this happens, but it happened to me. I thought the laptop was dead, which I got a new one, and I decided to toy around with the old one and it worked again. I was able to replicate the "fault" and online sources claim that it's essential to repower the CMOS battery after a complete disassembly. Some have PCB pads that let you do it, simply put a screwdriver on the pads to complete the circuit.

https://steamcommunity.com/discussions/ ... 760/?ctp=1
https://www.quora.com/What-happens-if-I ... ry?share=1
Unsure why mine didn't boot at all though.

mjkzz · Post by **mjkzz** » Sun Apr 12, 2020 6:07 am

Chris S. wrote:
mjkzz wrote:Yeah, Prime95 is the one.

What happened was, just to share it, I had a cool looking CPU heat sink from another project, decided to put it on the i7, the mistake was I simply added some thermal paste over existing leftover (semi-dried up) . . . then it started to crash whenever I ran Zerene

After Prime95 advised by Rik, I took the CPU out, cleaned its surface thoroughly, applied very thin layer of thermal paste, it never crashed again.
When PCs crash while running hard with stable software, the cause is almost always heat build-up. One of the most common causes of heat build-up is a problem with the thermal paste between the CPU and CPU cooler. The fix for this is thoroughly cleaning off any existing paste, finger prints, dust, or even a stray hair--getting CPU and heat sink utterly, scrupulously clean with (in my case) acetone and PEC pads. Then applying a very thin layer of new, quality thermal paste, and putting everything back together. Later, if the CPU-cooler contact needs to be broken for any reason, the old paste should be cleaned off thoroughly, and a new layer applied.

Thanks for those advice. I wish I knew it before . . . this is why I think I need to share my failure or point out something that seems trivial on the bases that it might be trivial to some, but unknown to others