[EONA]

Hello,

Last night I launched 4 cooperative jobs on the Maxwell Network, after the first render (Job 0) was finished, it showed "running" for Job 1 with "rendering" in all the 5 nodes, however the Time Left was empty, Current SL always 0 and Info "pending"... I didn't touch anything and left it running the whole night to see what would happen, and this morning it was still on the same way.

Another issue I had was the quality of the final image, the same render, from the same scene file with the same SL (12) is less grainy when I launch it locally than when done with Maxwell Network.

Can anyone help me?

Below is the Maxwell Manager log:

[20/May/2010 20:48:24] Monitor found
Connection with monitor enabled.
[20/May/2010 20:48:24] Monitor added to list:
Name: 10.0.0.116
Address: 10.0.0.116
Adding new job to queue..
[20/May/2010 20:50:32] ##### Getting next pending job #####
[20/May/2010 20:50:32] ##### Processing next pending job #####
Processing cooperative job: 0
Sending job order to node: WS06
Sending job order to node: WS25
Sending job order to node: WS23
Sending job order to node: WS03
[20/May/2010 20:50:32] ##### Getting next pending job #####
Render node: WS06: file_dependency_sent: 0649-100520_D04.mxs
Render node: WS06: file_dependency_sent: Teak01.jpg
Render node: WS06: file_dependency_sent: TeakplankLT60_6Strait-90V2.jpg
Render node: WS06: file_dependency_sent: anisotrpy1.png
Render node: WS06: file_dependency_sent: anisotrpy1.png
Render node: WS06: file_dependency_sent: metal_scratch.jpg
Render node: WS06: file_dependency_sent: normal_map.jpg
Render node: WS06: file_dependency_sent: Cayman_Islands_Ensign.png
Render node: WS06: file_dependency_sent: Cayman_Islands_Ensign.png
Render node: WS06: file_dependency_sent: fabric_TAP_RED.jpg
Render node: WS06: file_dependency_sent: fabric_TAP_Transmit.jpg
Render node: WS06: file_dependency_sent: fabric_TAP_BUMP.jpg
Render node: WS06: file_dependency_sent: agua01 copia.jpg
Render node: WS06: file_dependency_sent: agua04.jpg
Render node: WS06: file_dependency_sent: agua02 copia.jpg
Render node: WS06: file_dependency_sent: FINE DENIM.jpg
Render node: WS06: file_dependency_sent: FINE DENIM.jpg
Render node: WS06: file_dependency_sent: pB_granite_yellow_aple_512x512_d.jpg
Render node: WS06: file_dependency_sent: pB_granite_yellow_aple_512x512_b.jpg
Render node: WS06: file_dependency_sent: pB_granite_yellow_aple_512x512_b.jpg
Render node: WS06: file_dependency_sent: 092teck.jpg
Render node: WS06: file_dependency_sent: 092teck.jpg
Render node: WS06: file_dependency_sent: FINE DENIM BUMP.jpg
Render node: WS06: file_dependency_sent: metal+lattice+01_d100.jpg
Render node: WS06: file_dependency_sent: metal+lattice+01_o100.jpg
Render node: WS06: file_dependency_sent: metal+lattice+01_b015.jpg
Render node: WS06: file_dependency_sent: metal+lattice+01_s100+g100.jpg
Render node: WS06: file_dependency_sent: metal+lattice+01_w.jpg
Render node: WS06: file_dependency_sent: WeaveBump4.jpg
Render node: WS06: file_dependency_sent: WeaveBump2.jpg
Render node: WS06: file_dependency_sent: openweave02.jpg
Render node: WS06: file_dependency_sent: openweave02.jpg
Render node: WS06: file_dependency_sent: blue&white-canvas-color.jpg
Render node: WS06: file_dependency_sent: RADSKY-017.hdr
Render node: WS06 Job ID: 0 sl: 0.519331
Render node: WS25 Job ID: 0 sl: 0.519402
Render node: WS06 Job ID: 0 sl: 1
Render node: WS25 Job ID: 0 sl: 1
Render node: WS06 Job ID: 0 sl: 2.00001
Render node: WS25 Job ID: 0 sl: 2.00005
Render node: WS25 Job ID: 0 sl: 3.00005
Render node: WS06 Job ID: 0 sl: 3.00007
Render node: WS25 Job ID: 0 sl: 4
Render node: WS06 Job ID: 0 sl: 4.00094
Render node: WS03 Job ID: 0 sl: 0.519143
Render node: WS25 Job ID: 0 sl: 5.00001
Render node: WS03 Job ID: 0 sl: 1
Render node: WS06 Job ID: 0 sl: 5.00063
Render node: WS03 Job ID: 0 sl: 2.00207
Render node: WS25 Job ID: 0 sl: 6.00001
Adding new job to queue..
[20/May/2010 21:05:20] ##### Getting next pending job #####
[20/May/2010 21:05:20] ##### Processing next pending job #####
Processing cooperative job: 0
[20/May/2010 21:05:20] ##### Getting next pending job #####
Render node: WS03 Job ID: 0 sl: 3.0014
Render node: WS06 Job ID: 0 sl: 6.00001
Render node: WS25 Job ID: 0 sl: 7
Adding new job to queue..
[20/May/2010 21:10:36] ##### Getting next pending job #####
[20/May/2010 21:10:36] ##### Processing next pending job #####
Processing cooperative job: 0
[20/May/2010 21:10:36] ##### Getting next pending job #####
Adding new job to queue..
[20/May/2010 21:11:05] ##### Getting next pending job #####
[20/May/2010 21:11:05] ##### Processing next pending job #####
Processing cooperative job: 0
[20/May/2010 21:11:05] ##### Getting next pending job #####
Render node: WS23 Job ID: 0 sl: 0.51921
Render node: WS23 Job ID: 0 sl: 1
Render node: WS03 Job ID: 0 sl: 4.00002
Render node: WS23 Job ID: 0 sl: 2.00208
Render node: WS06 Job ID: 0 sl: 7.00028
Render node: WS23 Job ID: 0 sl: 3.00139
Render node: WS25 Job ID: 0 sl: 8
Render node: WS03 Job ID: 0 sl: 5.00062
Render node: WS23 Job ID: 0 sl: 4.00002
Render node: WS06 Job ID: 0 sl: 8
Render node: WS25 Job ID: 0 sl: 8.75643
Render node: WS23 Job ID: 0 sl: 5
Render node: WS03 Job ID: 0 sl: 6.00042
Render node: WS25 Job ID: 0 sl: 9
Render node: WS06 Job ID: 0 sl: 8.60528
Render node: WS23 Job ID: 0 sl: 6.00042
Render node: WS03 Job ID: 0 sl: 6.54254
Render node: WS25 Job ID: 0 sl: 9.52801
Render node: WS06 Job ID: 0 sl: 9
Render node: WS23 Job ID: 0 sl: 6.56248
Render node: WS03 Job ID: 0 sl: 7.00028
Render node: WS25 Job ID: 0 sl: 10
Render node: WS23 Job ID: 0 sl: 7.00001
Render node: WS06 Job ID: 0 sl: 9.54374
Render process finished in node: WS06 Job ID: 0
Starting MXI transference..
Render node: WS25 Job ID: 0 sl: 10.18
Render process finished in node: WS25 Job ID: 0
Starting MXI transference..
Render node: WS23 Job ID: 0 sl: 7.08904
Render process finished in node: WS23 Job ID: 0
Starting MXI transference..
Node WS06 has sent MXI file.
Render node: WS03 Job ID: 0 sl: 7.28737
Node WS25 has sent MXI file.
Node WS23 has sent MXI file.
Render process finished in node: WS03 Job ID: 0
Starting MXI transference..
Node WS03 has sent MXI file.
[20/May/2010 21:54:43] ##### Job finished. id: 0 #####
[20/May/2010 21:54:43] ##### Getting next pending job #####
[20/May/2010 21:54:43] ##### Processing next pending job #####
Processing cooperative job: 1
Sending job order to node: PCESPEN-006
Sending job order to node: WS06
Sending job order to node: WS25
Sending job order to node: WS23
Sending job order to node: WS03
Start merging process...
Merging process finished successfully!
[20/May/2010 21:55:16] Final MXI is not a network path and will be saved in a local path of the manager computer
[20/May/2010 21:55:16] Final MXI saved in: C:\TEMP\0649-100520_D04.mxi
[20/May/2010 21:55:17] Final image is not a network path and will be saved in a local path of the manager computer
[20/May/2010 21:55:17] Final image saved in: C:\TEMP\0649-100520_D04.jpg
[21/May/2010 09:13:42] Render node disconnected: WS23
[21/May/2010 09:51:08] Render node disconnected: WS06
[21/May/2010 10:48:06] Render node disconnected: WS03

[fuso]

Not sure if I can really help but I have come accross the same problem many times recently. It seems like the merging has
failed and therefore the cooperative job is the result of one local machine only. Have you tried to merge them manually? You
have to collect all the mxi's from whatever client has been part of the coop job, rename them, copy them into a local folder
and then merge them from there.

It seems the render network, especially the cooperative mode is rather buggy in the current release so I have temporarily
rolled back to V2.0.2. Unfortunately it is not much better either. But I never had any problems with the paths, you better
check them. Oh and why not using a better quality image format than jpg?

In case a coop job doesn't appear to run smoothly and one or more clients are hanging, I normally force to stop it and send
it again. I know, it's a pain...

[EONA]

I didn't try to merge them manually, but it wouldn't help in our situation, because the idea of our use of the network is to make it easier to launch many renders to be calculated using all the computers of the network. If we have to get 10 pieces of each render and merge them manually, at the end of the day it would be easier not to use the network, just launch each render on single machines.

The path is not a problem, just a warning, at the end it saves the MXI and the JPG in the correct place. I'm using JPG because we don't need more and everybody knows it.

We cannot force the renders to stop and send it again because we want to launch them during the night or weekends, so it wouldn't be anyone looking if they're frozen or not.

[polynurb]

hi EONA,

got similar issues with 2.0.3 with my setup.

no straight forward fix for it right now... what i do atm is rendering via command line using the "-node" flag, so i can use all licenses to start rendering locally from the nodes.

but if you render 4 jobs over night, it is really better to render these on 4 machines in parallel, and not 4 coop jobs after each other, there is no gain in speed.

the difference in grain is strange if it is not a coop render... haven't noticed anything like that.

[EONA]

Hi polynurb,
The idea is not to send X jobs in X nodes, but in 3*X nodes, so I'd really need the cooperative job to make them faster.
Anyway, I tried to launch through Maxwell Network single jobs, I put 7 jobs to run in 3 nodes, and what happened was that accordingly to the Maxwell Monitor none of the first 3 jobs of the stack finished, all 3 crashed betwen 60 and 95% and the other 4 didn't start. But when I went locally, I saw that all 3 renders were done and saved on the temp folder.
I also did many tests with the cooperative job the last days because regular Maxwell (no network) crashes when launching big renders with Multilight. So I launched these renders as cooperative jobs and what happens is that the Manager crashes every single time, but the nodes continue the renders... And then we have to get the files and merge.
Does anyone have an idea on how to solve these problems? Any patch to come?
Thanks!

[polynurb]

maybe that is then only a problem of the computer running the manager...

atm. it requires very much ram on the computer running manager to merge the mxi files.

i managed with ~ 3500 x 2000 and 9 multilight channel, but that is about the maximum a 8GB/64bit machine can handle; assuming that no other process eats up ram besides maxwell.

but you can always retrieve the mxi. from the nodes manually.. and merge by hand, before you try to merge with manager.. just to stay on the sure side!

[EONA]

Actually it crashes before it starts the merging... And the nodes continue to render and the only error message I get on the nodes is that they're not receiving a response from the manager when they try to send the mxis.

Next time I'll try to put the manager on a different computer to see what happens.

[EONA]

Small files and manager on a different and powerful computer, now the mximerger crashed. Log below:



Render node: WS23 Job ID: 0 sl: 7.0002
Render node: WS03 Job ID: 0 sl: 15
Render process finished in node: WS03 Job ID: 0
Starting MXI transference..
Node WS03 has sent MXI file.
Render node: WS23 Job ID: 0 sl: 8.00009
[28/May/2010 21:30:02] ##### Job finished. id: 0 #####
[28/May/2010 21:30:02] ##### Getting next pending job #####
[28/May/2010 21:30:02] ##### Processing next pending job #####
Processing cooperative job: 1
Sending job order to node: WS23
Sending job order to node: WS03
Start merging process...
Render process finished in node: WS23 Job ID: 1
Starting MXI transference..
Node WS23 has sent MXI file.
[28/May/2010 21:30:39] Error in merging process. The process crashed some time after starting successfully.
[28/May/2010 21:30:39] Merge process could not be completed. Temporary files are saved and can be merged manually.
[28/May/2010 21:30:39] Go to Menu/Open Temp Folder.
[28/May/2010 21:30:39] Merging process crashed!. Exit code: -1073741819
[28/May/2010 21:30:39] Temporary files have been saved in the manager temp folder and can be merged manually.
[28/May/2010 21:30:39] Temp folder:C:\DOCUME~1\xxx\LOCALS~1\Temp\mxnetwork\manager\job_0/