[limbus]

Hi there,
I was wondering what hardware components have the biggest influence on merging speed? CPU? Harddrive?

Our render manager does have enough RAM so that is not a bottleneck. Merging speed is usually not a problem but when we need to get a render out quickly and use all 20 or so nodes for rendering, the merging process can take quite long (2h or more). Now we are looking into ways to speed this up. Any tips or hints would be greatly appreciated.

Florian

[numerobis]

The fastest way seems to be manual merging with imput and output path on two different SSDs and a high clocked CPU.

If the network speed is not the problem and you are speaking about the time after 100% of the files have been copied, then one problem is, that the merging seems to be still single threaded.
And the other, that you can only specify a different output path for the manual merging...

http://www.maxwellrender.com/forum/view ... =2&t=39105

[limbus]

The fastest way seems to be manual merging with imput and output path on two different SSDs and a high clocked CPU.

I don't really want to go back to merging manually. But I guess adding one or two (as a RAID) SSDs might help anyway.

If the network speed is not the problem and you are speaking about the time after 100% of the files have been copied, then one problem is, that the merging seems to be still single threaded.
And the other, that you can only specify a different output path for the manual merging...

http://www.maxwellrender.com/forum/view ... =2&t=39105

Our network speed will be increased anyway so I am looking at ways to speed up the process after transferring the files to the render manager.
Thanks for the link.

Cheers, Florian

[dariolanza]

Hello limbus,

Ther merging process could be a bit slow not just for the merging process itself, but because it is performed by one single computer, and it merges all the files one after another (merges render1 and 2, then takes the resulting MXI and merges it with render3, etc).

It would be faster if instead of a linear process, it would be a tree-pyramid process, meaning that a node takes render1 and 2, merges them and sends it to the next node, with merges it with its current render, sends it to the next node, etc
Instead of merging every render in the same computer, having the nodes merging renders simultaneously and sending to the next node.
Merging simultaneously will finish the process in a fraction of the time.

Unfortunately this is not a feature that Maxwell could do today, but maybe you could manage your computers to manually implement a tree system like that.

Greetings

Dario Lanza
Render experts

[limbus]

It would be faster if instead of a linear process, it would be a tree-pyramid process, meaning that a node takes render1 and 2, merges them and sends it to the next node, with merges it with its current render, sends it to the next node, etc
Instead of merging every render in the same computer, having the nodes merging renders simultaneously and sending to the next node.
Merging simultaneously will finish the process in a fraction of the time.

Thanks for the info. I will think about this.

Cheers, Florian

[lifeofdave]

Hello limbus,

Ther merging process could be a bit slow not just for the merging process itself, but because it is performed by one single computer, and it merges all the files one after another (merges render1 and 2, then takes the resulting MXI and merges it with render3, etc).

It would be faster if instead of a linear process, it would be a tree-pyramid process, meaning that a node takes render1 and 2, merges them and sends it to the next node, with merges it with its current render, sends it to the next node, etc
Instead of merging every render in the same computer, having the nodes merging renders simultaneously and sending to the next node.
Merging simultaneously will finish the process in a fraction of the time.

Unfortunately this is not a feature that Maxwell could do today, but maybe you could manage your computers to manually implement a tree system like that.

Greetings

Dario Lanza
Render experts

Dario, in theory would this be the most efficient way to merge? There's a chance that simultaneous network transfers would create a bottleneck I suppose..

[limbus]

Dario, in theory would this be the most efficient way to merge? There's a chance that simultaneous network transfers would create a bottleneck I suppose..

Network speed could be a bottleneck. But I guess merging this way on one machine with multiple cores, enough RAM and a fast SSD would already speed things up.
Still thinking about how this could be automated.

Cheers, Florian

[lifeofdave]

Merging with multiple threads on one machine would certainly help, but what I meant in the quick diagram was some like this, merge nodes 1 and 2 on node 1, nodes 3 and 4 on node 3, nodes 5 and 6 on node 5 and so then. Then next round merge product of 1 and 2 with product of 3 and 4 on node 1, merge product of 5 and 6 with 7 and 8 on node 5 and so on.

Python scripting could be one way to automate it, wouldn't be simple though..

[limbus]

...but what I meant in the quick diagram was some like this, merge nodes 1 and 2 on node 1, nodes 3 and 4 on node 3, nodes 5 and 6 on node 5 and so then. Then next round merge product of 1 and 2 with product of 3 and 4 on node 1, merge product of 5 and 6 with 7 and 8 on node 5 and so on.

Thats how I understood Dario.

Python scripting could be one way to automate it, wouldn't be simple though..

Guess I gotta get started on those Phython tutorials. Something like Hazel (http://www.noodlesoft.com, Mac only unfortunately) could probably work as well together with some scripts.

[numerobis]

It would be faster if instead of a linear process, it would be a tree-pyramid process, meaning that a node takes render1 and 2, merges them and sends it to the next node, with merges it with its current render, sends it to the next node, etc .

So where is the bottleneck if only one core is used? Is it the hdd/ssd speed?
If it is the CPU, why can't this "cascaded" merging be done automatically on one machine, if the merging process itself can't be multithreaded?
I think copying the files multiple times from one node to another could be slower.

[dariolanza]

Hi,

In order to clarify my proposition, I've created these two schemes to let you see.

The first one describes the Maxwell default procedure. As you can see, all MXI must be sent to the Manager (16 in this example) that after receiving the last one starts merging them all one by one.

The second scheme describes the procedure I'm suggesting you to see if you have a way to implement in your local place.
As merges could be done in certain computers simultaneously, you can reduce the number of merging steps to 4, and even reduce the amount of simultaneous sends to a half (when maximum).

But aside from a custom programmed batch or so, I guess it can only be done locally by hand.

Cheers

Dario Lanza
Render expert