[Q2]

hi guys, new year, new problems.

sending a coop job to my nodes renders the single files and after completion each node sends the rendered mxi's to be merged. Thing is that all nodes eventually crash and the before rendered files get deleted in the Maxwell 4 Manager Temp folder, and the next coop render starts. All files of the first coop rendering are gone and the process starts allover again.

any ideas why?

here's the log file of one of the nodes:

[04/January/2017 11:19:56] Message from render process: time_update 7044
[04/January/2017 11:19:56] Message from render process: new_sampling_level_reached 9.886712
[04/January/2017 11:20:06] Message from render process: time_update 7054
[04/January/2017 11:20:06] Message from render process: new_sampling_level_reached 9.886712
[04/January/2017 11:20:16] Message from render process: time_update 7064
[04/January/2017 11:20:16] Message from render process: new_sampling_level_reached 9.886712
[04/January/2017 11:20:26] [INFO]: End writing MXI file...
[04/January/2017 11:20:26]
[04/January/2017 11:20:26] Message from render process: end_writing_mxi
[04/January/2017 11:20:26] Message from render process: time_update 7073
[04/January/2017 11:20:26] Message from render process: new_sampling_level_reached 9.886712
[04/January/2017 11:20:30] [INFO]: Start writing image file...
[04/January/2017 11:20:30] Message from render process: start_writing_image
[04/January/2017 11:20:30]
[04/January/2017 11:20:36] Message from render process: time_update 7083
[04/January/2017 11:20:36] Message from render process: new_sampling_level_reached 9.886712
[04/January/2017 11:20:38] [INFO]: End writing image file...
[04/January/2017 11:20:38]
[04/January/2017 11:20:38] [04/January/2017 11:20:38] [INFO]: Benchmark of 293.747. Time: 1h58m07s. SL of 9.89
[04/January/2017 11:20:38] Message from render process: end_writing_image
[04/January/2017 11:20:38] Message from render process: time_update 7087
[04/January/2017 11:20:38] Message from render process: new_sampling_level_reached 9.88671
[04/January/2017 11:20:38] Message from render process: render_info rendering...6 percent
[04/January/2017 11:20:38] [04/January/2017 11:20:38] [INFO]: Time left: 28h34m23s
[04/January/2017 11:20:46] Message from render process: time_update 7095
[04/January/2017 11:20:46] Message from render process: new_sampling_level_reached 9.890121
[04/January/2017 11:20:56] Message from render process: time_update 7105
[04/January/2017 11:20:56] Message from render process: new_sampling_level_reached 9.894483
[04/January/2017 11:21:06] Message from render process: time_update 7115
[04/January/2017 11:21:06] Message from render process: new_sampling_level_reached 9.899708
[04/January/2017 11:21:16] Message from render process: time_update 7125
[04/January/2017 11:21:16] Message from render process: new_sampling_level_reached 9.904056
[04/January/2017 11:21:26] Message from render process: time_update 7135
[04/January/2017 11:21:26] Message from render process: new_sampling_level_reached 9.908394
[04/January/2017 11:21:36] Message from render process: time_update 7145
[04/January/2017 11:21:36] Message from render process: new_sampling_level_reached 9.913589
[04/January/2017 11:21:46] Message from render process: time_update 7155
[04/January/2017 11:21:46] Message from render process: new_sampling_level_reached 9.917911
[04/January/2017 11:21:56] Message from render process: time_update 7165
[04/January/2017 11:21:56] Message from render process: new_sampling_level_reached 9.922225
[04/January/2017 11:22:06] Message from render process: time_update 7175
[04/January/2017 11:22:06] Message from render process: new_sampling_level_reached 9.927395
[04/January/2017 11:22:16] Message from render process: time_update 7185
[04/January/2017 11:22:16] Message from render process: new_sampling_level_reached 9.931691
[04/January/2017 11:22:26] Message from render process: time_update 7195
[04/January/2017 11:22:26] Message from render process: new_sampling_level_reached 9.936838
[04/January/2017 11:22:36] Message from render process: time_update 7205
[04/January/2017 11:22:36] Message from render process: new_sampling_level_reached 9.941117
[04/January/2017 11:22:46] Message from render process: time_update 7215
[04/January/2017 11:22:46] Message from render process: new_sampling_level_reached 9.945393
[04/January/2017 11:22:56] Message from render process: time_update 7225
[04/January/2017 11:22:56] Message from render process: new_sampling_level_reached 9.949662
[04/January/2017 11:23:06] Message from render process: time_update 7235
[04/January/2017 11:23:06] Message from render process: new_sampling_level_reached 9.954770
[04/January/2017 11:23:16] Message from render process: time_update 7245
[04/January/2017 11:23:16] Message from render process: new_sampling_level_reached 9.958935
[04/January/2017 11:23:26] Message from render process: time_update 7255
[04/January/2017 11:23:26] Message from render process: new_sampling_level_reached 9.963177
[04/January/2017 11:23:36] Message from render process: time_update 7265
[04/January/2017 11:23:36] Message from render process: new_sampling_level_reached 9.967160
[04/January/2017 11:23:46] Message from render process: time_update 7275
[04/January/2017 11:23:46] Message from render process: new_sampling_level_reached 9.972067
[04/January/2017 11:23:56] Message from render process: time_update 7285
[04/January/2017 11:23:56] Message from render process: new_sampling_level_reached 9.976117
[04/January/2017 11:24:06] Message from render process: time_update 7295
[04/January/2017 11:24:06] Message from render process: new_sampling_level_reached 9.979994
[04/January/2017 11:24:16] Message from render process: time_update 7305
[04/January/2017 11:24:16] Message from render process: new_sampling_level_reached 9.984704
[04/January/2017 11:24:26] Message from render process: time_update 7315
[04/January/2017 11:24:26] Message from render process: new_sampling_level_reached 9.988821
[04/January/2017 11:24:36] Message from render process: time_update 7325
[04/January/2017 11:24:36] Message from render process: new_sampling_level_reached 9.992680
[04/January/2017 11:24:46] Message from render process: time_update 7335
[04/January/2017 11:24:46] Message from render process: new_sampling_level_reached 9.997199
[04/January/2017 11:24:53] [INFO]: Start writing MXI file...
[04/January/2017 11:24:53]
[04/January/2017 11:24:53] Message from render process: start_writing_mxi
[04/January/2017 11:24:56] Message from render process: time_update 7345
[04/January/2017 11:24:56] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:06] Message from render process: time_update 7355
[04/January/2017 11:25:06] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:16] Message from render process: time_update 7365
[04/January/2017 11:25:16] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:26] Message from render process: time_update 7375
[04/January/2017 11:25:26] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:36] Message from render process: time_update 7385
[04/January/2017 11:25:36] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:46] Message from render process: time_update 7395
[04/January/2017 11:25:46] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:25:56] Message from render process: time_update 7404
[04/January/2017 11:25:56] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:06] Message from render process: time_update 7414
[04/January/2017 11:26:06] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:11] TCP message from manager received.
[04/January/2017 11:26:11] Message from manager: force_stop_render
[04/January/2017 11:26:11]
[04/January/2017 11:26:11] Render stopped!!
[04/January/2017 11:26:11] [INFO]: Stopping render
[04/January/2017 11:26:11]
[04/January/2017 11:26:16] Message from render process: time_update 7424
[04/January/2017 11:26:16] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:26] Message from render process: time_update 7434
[04/January/2017 11:26:26] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:36] Message from render process: time_update 7444
[04/January/2017 11:26:36] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:46] Message from render process: time_update 7454
[04/January/2017 11:26:46] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:26:56] Message from render process: time_update 7463
[04/January/2017 11:26:56] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:06] Message from render process: time_update 7473
[04/January/2017 11:27:06] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:16] Message from render process: time_update 7483
[04/January/2017 11:27:16] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:26] Message from render process: time_update 7493
[04/January/2017 11:27:26] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:36] Message from render process: time_update 7503
[04/January/2017 11:27:36] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:47] Message from render process: time_update 7513
[04/January/2017 11:27:47] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:27:57] Message from render process: time_update 7522
[04/January/2017 11:27:57] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:07] Message from render process: time_update 7532
[04/January/2017 11:28:07] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:15] [INFO]: End writing MXI file...
[04/January/2017 11:28:15]
[04/January/2017 11:28:15] Message from render process: end_writing_mxi
[04/January/2017 11:28:17] Message from render process: time_update 7542
[04/January/2017 11:28:17] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:20] [INFO]: Start writing image file...
[04/January/2017 11:28:20] Message from render process: start_writing_image
[04/January/2017 11:28:20]
[04/January/2017 11:28:27] Message from render process: time_update 7552
[04/January/2017 11:28:27] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:37] Message from render process: time_update 7562
[04/January/2017 11:28:37] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:47] Message from render process: time_update 7572
[04/January/2017 11:28:47] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:57] Message from render process: time_update 7582
[04/January/2017 11:28:57] Message from render process: new_sampling_level_reached 10.000040
[04/January/2017 11:28:58] [INFO]: End writing image file...
[04/January/2017 11:28:58]
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: Benchmark of 287.531. Time: 2h06m27s. SL of 10.00
[04/January/2017 11:28:58] Message from render process: end_writing_image
[04/January/2017 11:28:58] Message from render process: time_update 7587
[04/January/2017 11:28:58] Message from render process: new_sampling_level_reached 10.00004
[04/January/2017 11:28:58] Message from render process: render_info rendering...7 percent
[04/January/2017 11:28:58]
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: Time left: 25h48m48s
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: Ending render process..
[04/January/2017 11:28:58]
[04/January/2017 11:28:58] [INFO]:
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: Render finished successfully
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: -nowait flag enabled. Closing app..
[04/January/2017 11:28:58] [04/January/2017 11:28:58] [INFO]: Ending Session...
[04/January/2017 11:29:17] The remote host closed the connection . Code: 1
[04/January/2017 11:29:18] ERROR: Error in rendering process. The process crashed some time after starting successfully.
[04/January/2017 11:29:18] ERROR: Render process crashed!
[04/January/2017 11:29:18] Connecting to render process: Binding to port: 45463
[04/January/2017 11:29:21] TCP message from manager received.
[04/January/2017 11:29:21] Message from manager: cpuid
[04/January/2017 11:29:21] 26
[04/January/2017 11:29:21] New job order received

CHEERS

Q!

[Q2]

More Tests, More Problems:

tried using the old manager and monitor (3.2.1.5) which used to work despite the error messages, but that ship has sailed too. now it does not work either.

here's the error Messages I get fromm each rendernode:

[11/January/2017 11:02:37] [INFO]:
[11/January/2017 11:02:37] [11/January/2017 11:02:37] [INFO]: Render finished successfully
[11/January/2017 11:02:37] [11/January/2017 11:02:37] [INFO]: -nowait flag enabled. Closing app..
[11/January/2017 11:02:37] [11/January/2017 11:02:37] [INFO]: Ending Session...
[11/January/2017 11:02:50] ERROR: Error in rendering process. The process crashed some time after starting successfully.
[11/January/2017 11:02:50] ERROR: Render process crashed!
[11/January/2017 11:02:50] Connecting to render process: Binding to port: 45463

ANYONE have an idea why all of a sudden the nodes keep crashing, despite the fact that they all render fine and juat after completion they evaporate! ?

Rendering the same Scene local finishes fine... so my guess is that the nodes get screwed up by sending the finished files to be merged... maybe, or. aehm, du now???

[Mihai]

You don't have the rendered files on each nodes temp folder so you can recover the render? They should not get deleted if they are not successfully transferred to the computer that's running the Manager.

[Q2]

I found the culprit!

IF an emitter filename is LONGER than 30 characters the nodes crash!

When the filename is shorter, all nodes are rendering fine and the files get merged all right!

PUH!

That cost me some nerves... do I get an free espresso form NL for finding this out?

Cheers

Q!

[Q2]

Mihai

You are correct, the rendered files are in each nodes temp folder, but copying them all and merging that by hand is a big pain in the bottox if you have to render a lot...

[Mihai]

Sure, I understand. Just thought it was worth mentioning in case.

Very weird issue though...yes you should get a bug hunting prize for this one