[Kevron]

Hi and a happy new year to all maxwell users

I have rendered a sequence over the holiday across our local farm, and I am looking for some advice to help me find a stable solution.

My problems is that the rendering just stops after a random amount of frames. If I set 0 - 660 as desired frames, the farm will render sometimes 0 - 270. Then I'll restart the network and start from 271-660. Then it would render as normal before it suddenly stops at 354. And so it goes on. 355-660, 443-660, 570-660.

I am trying to see it the log could tell me something, but it states that the manager is transmitting new frames to the network. So as far as I can tell, there is no real error message for me to decipher.

I have tried to render on a only Mac based farm with send dependancies. And it shows the same behavior as when I mixed up the network with a pc.

I use the latest maxwell on all nodes and mountain lion on the macs.

Do any of you have any clues on how I could find out what really happens when the rendering just stops? Just chime in, and if you need more information about my setup I'll happily share it with you.

Best regards,
Tor Martin

[Mihai]

Happy new year Tor!

Some questions:
How many nodes are you rendering on?
Same OS version and hardware on all? How much RAM? Possibility that you are running out of temp disk space?

Try turning on deep debug mode on all nodes plus manager and monitor, this might help give a clue why rendering stops.

[Kevron]

Hi Mihai,
I am only rendering on four nodes. Three has the same OS. Hardware differ between all four. (one iMac, the rest is Mac pro's) Ram on the manager machine is 48, nodes has 36, 24 and 8.

Temp disk is a SSD drive (350 gb available).

Did not know about deep debug mode, have activated it now on all nodes, manager and monitor. Also started a new render to see if the log will give me some new information.

I'll post when I find something new to share.

Thank you for your answer!

[Bubbaloo]

Sometimes I have found that the monitor will stop responding, but the renders are still going. Try closing the monitor, and reopening it and see if it updates.

[Kevron]

Thanks for the tip Brian, I'll sure try that also the next time it seems to stop. The render is still going, so nothing new to report yet.

Regards,
Tor M

[Kevron]

After rendering to SL 14 (0-660), it just stopped rendering at frame 43. Here is a screengrab from the monitor showing what I see.


And here are all the logs created by all the nodes, manager and monitor:
https://dl.dropbox.com/u/1059954/LieOgH ... t/Logs.zip

I tried Brian´s tip on closing and reopen the monitor, thats why the warning appears in the log "warning: Monitor disconnected". But I did this after the rendering already had stopped.

So, could some smarter minds take a look and see if there is a explanation to this in the logs?

edit: After starting another render, I see that the attached logs are not "deep debug". I guess I had to restart the Monitor, Manager and Nodes to get the change. So, I´ll upload some new logs next time the render stops.

Regards,
Tor Martin

[Mihai]

Any new logs uploaded?

[Kevron]

I have a render going now. Will post all the logs as soon as it stops. I see a lot of green text in the log now, so I guess I manage to enable deep debugging.

I wonder if the sudden stop in the render is network related. There are some huge files being created from maxwell, 660 frames gives me 200 gb of data. Off cause some of the blame is enabling intensity in multilights.

Maybe the network just can't cope with the amount of transfer from all the nodes?

Just speculating here, but I'll post the logs and hope that they could shine some lights.

Thanks for checking up on it!

[eric nixon]

Can you adjust the multilight intensity across an animation? If you can I guess you would need to set the multilight-keyframes before setting off the job.

[Kevron]

Hi Eric, I mainly use multi-light for giving me the extra possibilities in post. I do the post work in Nuke, and when multi-light is enabled I get all my lights nicely in separate layers in the exported exr files.
Here is a contact sheet that shows my passes for the dog:


Then it is easy to exclude parts from the light, but also color adjust it. So bluer rim lights, more red key light and so on. And I also convert the exported exr files to 16bit half float with 1 zip line compression. Works much faster in Nuke, and the files is down to 30% off its original size.

[Kevron]

HI, here are the new logs with deep debug on: https://dl.dropbox.com/u/1059954/LieOgH ... g_logs.zip

Job 1 (26-120) was successful, but Job 2 (121-221) stopped at frame 143.

Hope the logs can tell you something.

Best,
Tor Martin