[juan]

Hi Kevron,

Could you please give it another attempt with "send dependencies" unchecked"? We have detected this is the origin of most of the network problems and actually in v3 it is disabled by default.

Thanks,

[Kevron]

I can´t really go for "send dependencies" unchecked, there is not a single shared folder that is easy to access from every nodes. It is also a mixed network of OS.

If I quit and restart Maxwell Network Monitor (while the nodes are rendering), then I am able to stop a job. It seems like it "hangs" after a while.

I managed to find out why it did not render across the network, it was a plugin bug (c4d). But Network Monitor still "hangs".

Thanks,
Tor M

[Mihai]

So it's simply that Monitor hangs, but when you restart it, you see the jobs ok and they are rendering normally?

[Kevron]

Yep

[Kevron]

Hi guys, after several days of testing with Maxwell v3 - I have to sadly say that this is the least stable rendering solution for us.
We have a very small farm of just 5 (4 Mac and 1 PC) nodes, so it is important that they "do their job" 24/7.

I have both tried "send dependencies" and "pack and go". Their results is somewhat similar in stability.

Some of the problems:
- Maxwell Monitor hangs (You cant submit jobs, you cant preview jobs, cant stop etc..)
- There is ALOT of "Maxwell Render quits unexpectedly" (This happens also when rendering on a single machine. Usally when a render is finished, and the program should quit in a "normal" way)
- Our Pc node HAS to be started with mxnetwork.exe -monitor:xxx.xxx.x.xx. If you only uses the shortcut that Maxwell creates on the desktop, the "Maxwell Render Node" freezes.
- Same thing happens when you press "reset entire network" - it restarts the Mac nodes ok, but the Pc node goes into a "freeze" state. You then need to force quit it, and start it with mxnetwork.exe -monitor:xxx.xxx.x.xx
- Jobs has vanished from the Maxwell Monitor. My Pc node was rendering during the night, unexpectedly stopped, job gone from the monitor. (Was a animation job, progressive SL 6,10,16)
- The regular 3 of 5 nodes start to render the job, the two others dont. And its always random which node that starts and which node that does not. On a great day, all nodes work. But that is more and more rare.

I have no problem Maxwell being a slow renderer, because it creates beautiful images. But I tend to spend more and more hours trying to figure out a decent rendering pipeline.
It was quite stable in v 2.7, but I had to babysit it to watch out for errors.

How can I help you help me? What kind of debug data, scenes, network information could I give you so that you could help me sort this out. I just want to render, how hard could it really be.

Regards,
Tor M

[juan]

Hello Tor,

If the 2.7 network was stable the v3 one should be stable as well, the architecture is very similar with the addition of some extra layers to control errors. We have also updated the libraries that handle the communication with native sockets of the OS. Have you changed anything in your network configuration, in terms of machines involved, firewalls, antivirus, etc?

You can send us the log files of all the machines (manager, monitor(s) and nodes) and we will try to diagnose what is going on. Please send them through the customer portal form or post a download link here or via PM, anything is fine.

Thanks,

Juan

[Mihai]

- Our Pc node HAS to be started with mxnetwork.exe -monitor:xxx.xxx.x.xx. If you only uses the shortcut that Maxwell creates on the desktop, the "Maxwell Render Node" freezes.

This sounds very strange. You mean, you click the green Node icon on the desktop to start a Node right? Because that shortcut you wrote above would start a Monitor, not a node.

- There is ALOT of "Maxwell Render quits unexpectedly" (This happens also when rendering on a single machine. Usally when a render is finished, and the program should quit in a "normal" way)

This happens with all the scenes you've tried so far? Could it be memory issue? What render rez, many ML sliders? What is the RAM usage during render?

[Kevron]

Hi Juan and Mihai, I appreciate your help and guidance.

Your absolutely correct Mihai, it was a misspelling - what I really ment was:

Code: Select all

mxnetwork.exe -node:xxx.xxx.x.xx

I´ll try to collect at much data as possible to give you, here is the crash log that apple generates (not sure if is has any use to you, but the more you get, the better I reckon)
https://dl.dropboxusercontent.com/u/105 ... Report.rtf

Here is the MXI file that I am rendering : https://www.dropbox.com/s/eids8tb3fglun ... ht_v57.mxi
It is 6 ML sliders, displacement and Motion Blur (PLA). I cant use fast multilight preview (Error running GPU Multilight (-54): ).
Total rendertime to reach SL 18 is 6h47m26s. And the animation is 700+ frames, so this is not possible.
I have removed the background (rendered as a still) and rendering just a character pass. Removed also ML and Motion Blur. Now the frame is 1h30m on the fastest machine. (SL15) Which seems also a bit slow. But I reckon it is the SSS material that is the main reason for the excessive rendertime.

The network consist of 5 nodes. The "main" Mac is also the Manager and Monitor. Its OS X 10.8.5 2x2,66 GHZ &-Core Intel Xeon 48 GB 1333 MHZ DDR3
The other nodes:
OS X 10.8.5 - 2x 2,26 GHz QuadCore Intel Xeon 24 GB 1066 DDR3
OS X 10.8.5 - 2x 3,2 GHz QuadCore Intel Xeon 24 GB 800 MHz DDR2 FB-DIMM
OS X 10.9 - 2,8 GHz Intel cire i7 16 GB 1067 MHz DDR3
Windows 7 Professional - Intel Xeon CPU 2x 2,40Ghz 12,0 GB RAM

Since it is a mixed network, "send dependencies" has been the easiest thing to do. But I am trying now a shared folder on a shared network drive.

If the Mac´s path is: "Volumes/work/MaxwellR/S1/s1_Male/s1_Male_characterPass/"
And the Pc path is: "Z:\work\MaxwellR\S1\s1_Male\s1_Male_characterPass/"

What would a unified path be? So that both Mac and Pc could read and write to the same folder?

We are a small company consisting of three people, where non of us is especially tech-savvy when dealing with "this issues". So bear in mind that some stupid questions could occur on my hunt for successful rendering.

I will update this thread with logs on Monday.

Thanks,
Tor M

[Kevron]

Another crash report from another Mac with Maxwell 3: https://www.dropbox.com/s/r4xaipbhx5zg5 ... 20kopi.rtf

Regards,
Tor M

[juan]

Hi Tor,

Could you please also update the logs written by the network application? They are typically in the temp/mxnetwork folder, to get them just go to the main menu/"open temp folder", you will see the logs there, each node, monitor and manager saves a different log file. They will be very helpful to understand what is going on.

From what I have seen in the crash logs, it seems the render process is crashing itself, not the network. Could you please check if the scene renders locally in the same machine without using the network system?

Thanks!

Juan

[Kevron]

Hi Juan,

Here are the logs (5 nodes, manager and monitor): https://dl.dropboxusercontent.com/u/105 ... rkLogs.zip

Regarding the crashes. If I just open Maxwell and then quit it. 3 out of 10 times it will quit with a crash. So, it does it job - but crashes when quitting.

Another thing I have noticed with the network render. If you start the render with 3 nodes, and after a while you start up the other 2 nodes. The two last nodes will always give you a first rendered frame "error"

See attached image: https://dl.dropboxusercontent.com/u/105 ... artsUp.png

See f 113. Here a node was started up (192.168.1.152) which is Eskils-MacBook-Pro. But the node does not really do anything on this frame. Then the node gets a new frame, and the name goes from the IP address to the computer name - and it starts rendering.

F 79 is also the same. So the state will always be running.


Do you also have any guidance regarding this:

If the Mac´s path is: "Volumes/work/MaxwellR/S1/s1_Male/s1_Male_characterPass/"
And the Pc path is: "Z:\work\MaxwellR\S1\s1_Male\s1_Male_characterPass/"

What would a unified path be? So that both Mac and Pc could read and write to the same folder?

Any help would be appreciated!

[juan]

Hi Tor,

Thanks for updating the log files. We have been looking into them and all the crashes seem related to the render process itself, not the network system. All the errors are like this one:

Code: Select all

[09/December/2013 03:49:53] ERROR: Error in rendering process. The process crashed some time after starting successfully.

It seems it happens when writing the image, we will look into that further.

We will make more tests here, but it seems it can be scene specific. If that was the case, then that scene should crash also locally, just opening Maxwell (.exe or .app) and rendering.

Regarding the crashes. If I just open Maxwell and then quit it. 3 out of 10 times it will quit with a crash. So, it does it job - but crashes when quitting.

That is a clue, and actually can explain some of the problems above.

Thanks!

[juan]

Hi again Tor,

We have seen that all the crashes could be related due a problem saving image files, as explained here:

http://www.maxwellrender.com/forum/view ... 85#p374785

Please let us know if the issue persists with the new build.

Thanks,

[Kevron]

Hi Juan,

I updated maxwell on both Mac and PC nodes. The little farm has worked for 12 hours without the slightest error message. So it seems that the lates build fixed the issues. Thanks for your response and updates.

Regarding a unified path that both Mac and PC can read&write to, any tips on this? See some earlier posts on the different paths.

Regards,
Tor Martin

[juan]

Great to hear so Tor,

Regarding a unified path that both Mac and PC can read&write to, any tips on this? See some earlier posts on the different paths.

For read-only folders (i.e for reading textures), "send dependencies" is the recommended way to go in networks with different OS's (it is not necessary in networks where all the machines are running the same OS.).

For output folders where the images are written the situation depends on the OS that the manager is running. The golden rule is that the output path has to be in a format that the manager understands, because the manager is the one who writes the final output. If the manager does not understand the path, it saves the output in its temp folder so you can recover it later.

-if i.e the manager is a Mac machine, the output path cannot be in windows format, or it will fail. Also the output path must be a folder that is already mounted in the manager, otherwise the authentication process will fail because OSX internally will ask user/pass to get access to the folder. Therefore the user should mount the output paths in the manager when it starts, before launching the jobs.

-if the manager is a Windows machine, the output path should be in windows format (starting with "\\", or a mounted folder)

I hope it helps.