[gehany]

Hey guys,

Still trying to get my office render farm off the ground. I've run into the problem where after a few hours, my Maxwell Monitor becomes unresponsive. There isn't any good information in the logs...on the Monitor it still says the job is running and the nodes are rendering, but they aren't. The nodes have mixed messages, some say that the job was completed, others say that the manager closed and they are looking for a manager. I'd say this happens roughly at about SL 15, and I'm rendering to about 15-20 computers. Any tips on how to make the Monitor a little bit more responsive/robust? Thanks.

[gehany]

Additional information...Monitor and Maxwell are running on the same server, all mxi mxs files are hosted in an open network share hosted on that server, and I have a firewall exception for the server on all my machines.

[Bubbaloo]

Does the job finish? It sounds like the manager is crashing.

[gehany]

No it neer finished. You're right that it could be that manager is locking up. Manager is running on Server 2008 R2, same machine as Monitor.

[gehany]

I confirmed it: it is Manager that is locking up. Any way to fix this?

[Mihai]

Switch to deep debug mode on the Manager and check the logs. Could it be a RAM problem? Does the RAM usage increase as the render progresses?

[gehany]

So it appears that Manager isn't running out of RAM, but it is running out of CPU, and not taking advantage of multiple cores. Manager is maxing out one of the cores and then crashing. Please see the below picture.

[gehany]

Bump. Is there anyway to make Manager use multiple cores? I don't want to run out and purchase another computer to run manager if I can avoid it.

[Mihai]

There is something strange going on here, because Manager first of all shouldn't really use 100% of your CPU, it's just managing the renders, not calculating them. From the screenshot, it doesn't seem to be maxing out one core, but instead spiking from time to time which is normal (it's handling communications and updates between the nodes).

Is this computer only running Manager and Monitor, no node? You could actually try and run Manager on a machine that's running a node and see if it also crashes. I'm really not sure if this has something to do with Server 2008, but maybe it's the network writes that can be a cause of the crashes. Make sure you set the temp folders of each node to a local folder (you can do this from each nodes preferences from the File menu), NOT the network drive on this machine that's running the Manager/Monitor. Only the final MXI files should be transferred to this machine when the render is finished.

[gehany]

Ok, you would suggest running render node on the server, or running the manager on one of the render nodes? Also, will manager have problems connecting to a 20+ node job on a non-server OS? I know the amount of connections the computer can make is capped on a non-server OS.

[polynurb]

will manager have problems connecting to a 20+ node job on a non-server OS?

it is best to not have any non-server OS computer in the network which is accessed by +10 computers.
hard connection time outs, even if not related to maxwell network traffic, have a negative effect on network reliability.

i noticed that when upgrading beyond 10 nodes on our local lan.

also it could happen that other network activity blocks maxwell network traffis; eg. if your render repository computer is also a general file server accessed by workstations.

[gehany]

Ok, I had some success running mananger off of a virtual win7 machine. Manager still crashed occasionally, but have been able to complete some jobs.

HOWEVER, I am having trouble seeing all of my render nodes. I did push out a firewall exception for both mxnetwork and the ip address of the machine to the office. The error message I'm getting in Manager is:

[28/March/2013 12:12:01] getRenderNode address not found: 192.168.10.96
[28/March/2013 12:12:01] Receiving broadcast from node in port: 60993 at address 192.168.10.96

This is just one IP address, it is getting this error with many IP addresses. The IP addresses correlate to machines running render node that are not showing up in manager. Any ideas?

[gehany]

Disregard my previous post...seems like the render nodes are able to reconnect if you kill mxnetwork.exe and relaunch render node. Not ideal but I'll take it.



However, Manager continues to lock up: I have tried it on Win7,Server2008R2, and Server2012. I do have a dmp file from server2012 that I could upload if you think that it would help you diagnose the problems with Manager.