Page 1 of 1

All is well with 3.1. What about netowrk rendering?

Posted: Tue Dec 16, 2014 3:11 am
by gianca
Tested the new camera response, spot lights and briefly touched the render zones. No issues to report, all good!
:)

I recall Juan mentioning in the blog about the new network rendering: is that still happening?
I'd love to try that as I believe the current network rendering is Maxwell Achille's heel: just few weeks ago I lost a night of rendering because I dared to close the monitor before double checking that the reported as finished renders were done for real... Zero files found... Not even the temp (of course). That teaches me.
I usually use Deadline to manage long and time sensitive renders.

G

Re: All is well with 3.1. What about netowrk rendering?

Posted: Tue Dec 16, 2014 4:02 pm
by Mihai
Well, I apologize if this sounds like a silly question but you're sure it's not the Manager you closed? Because the Monitor only reports what the Manager tells it, and the Manager runs the show. Network rendering works without any Monitor, you would use it only to add jobs, then you can close it and reopen it when you like, it will receive its data from the Manager again.

But in any case, this new system is not fully ready for the 3.1 release and we will release a testing version of it soon, probably after the holidays. We still need to transfer over some of the features already present in the current system.

Re: All is well with 3.1. What about netowrk rendering?

Posted: Tue Dec 16, 2014 7:57 pm
by gianca
Not a silly question at all, as in the past it did happen to me to close the manager when the monitor said it was done but the manager did not actually finish to send the mxi file yet. Oops.
In this particular case i checked in the morning to be sure the monitor, the render nodes and the manager were done: since all the status were quiet I assumed it was done and I close all down but the final mxi file were missing. Gone. Even the previous mxi preview which I run hours earlier was nowhere to be found (of course maxwell delete *everything* after it closes no matter what you specify in the prefs). Not the first time I got into this: it is entirely possible in the past I did close one of 3 moving parts before checking so it may be my fault few times, but the bottom line is that I should follow this rule: never, *ever* close any of the apps till you check the actual file is actually done and written on disk, but if there is any smell of funky issues at *all* copy all the files from the temp directory before closing anything. In fact it is a safe procedure once in a while to backup files while it's still rendering, especially for the very long, expensive ones.
To explain what I consider "funky" it is any issue where the monitor or manager or render node is not doing what it has been directed to do, like reporting jobs as finished while it may be still rendering or merging, refusing to pick up jobs after being paused, not respecting the que order, render nodes not picking up any more jobs after finishing a render, unable to reset, unable to stop, etc.
I get problems like this randomly and they may be caused by network issues, but honestly after working with maxwell on 2 studios plus mine (which went trough at least 2 generations of routers, servers and render nodes) the only common denominator over all these issues seems to be the network rendering itself.

To be honest is really, really difficult to report these problems: I tried really hard in the past but it was time consuming and in the end next limit support could not understand were the problems where based on my reports, which is understandable without being on site.
I gave up on reporting years ago because of that but the problems are still there: sometimes I get lucky, sometimes I may do mistakes, but sometimes the network rendering catch me with the pants down. I' m still walking funny after last one. ;)
My solutions are to follow the rule above or use deadline: deadline or running renders manually (which is almost the same thing) never failed me.
So I'm looking forward to the next generation, which sounds very, very promising.

Gianca
Mihai wrote:Well, I apologize if this sounds like a silly question but you're sure it's not the Manager you closed? Because the Monitor only reports what the Manager tells it, and the Manager runs the show. Network rendering works without any Monitor, you would use it only to add jobs, then you can close it and reopen it when you like, it will receive its data from the Manager again.

But in any case, this new system is not fully ready for the 3.1 release and we will release a testing version of it soon, probably after the holidays. We still need to transfer over some of the features already present in the current system.

Re: All is well with 3.1. What about netowrk rendering?

Posted: Thu Dec 18, 2014 4:30 pm
by Mihai
In those cases you should always take a look at the Manager and Node logs (Monitor logs aren't so useful). Any previews you do will be stored in the temp folder of the computer running the Monitor. When you close it, yes, those previews are deleted. There can be errors also when merging all those MXI files if the computer that's running the Manager runs out of RAM in the case of merging very large MXI files and/or with many ML sliders. Manager will give you error messages about this though in its log window.

Re: All is well with 3.1. What about netowrk rendering?

Posted: Thu Dec 18, 2014 6:26 pm
by gianca
Mihai wrote:In those cases you should always take a look at the Manager and Node logs (Monitor logs aren't so useful). Any previews you do will be stored in the temp folder of the computer running the Monitor. When you close it, yes, those previews are deleted. There can be errors also when merging all those MXI files if the computer that's running the Manager runs out of RAM in the case of merging very large MXI files and/or with many ML sliders. Manager will give you error messages about this though in its log window.
Well, that's what I said it's my rule and I did check everything before closing down but it did not save me this time: monitor log said it was done, render node log was done, the manager was done, I did not see any errors reported in the logs, everything seemed to be ok, yet there was no files saved in the output directory.
It is entirely possible that the merging run out of memory as I'm greedy about my multilights, but if it was the error may have been so buried in the log I missed to detect it...
There are currently too many things to check if a render had issues which is why I found deadline or manually starting renders a more reliable workflow.
I'm hoping the new network rendering will make the whole process more robust and more transparent to issues.

G

Re: All is well with 3.1. What about netowrk rendering?

Posted: Thu Dec 18, 2014 8:22 pm
by gianca
BTW this gotta to be my worst choice of topic title ever... :(

Re: All is well with 3.1. What about netowrk rendering?

Posted: Thu Dec 18, 2014 8:28 pm
by Mihai
gianca wrote: I did not see any errors reported in the logs, everything seemed to be ok, yet there was no files saved in the output directory.
It can be also that Manager doesn't have enough permissions to save the final file in the directory you specified. But it will give you a warning in this case. It's not a red color text, but the same color except it starts with a big WARNING text label. You can set the logs so it only shows you warnings and errors. Then you know that if something is written there - would be interesting to see what it says :)