[zparrish]

1st off, this post "might" need moved to the network rendering forum, but I wanted to start here because it's heavily dependent on maxwell.exe and the Maxwell Shell Extension. If it is determined to be related to network rendering, then this can be "tabled" and brought back up once the new network tools are available. This issue happens on multiple nodes at the same time, so it very well could be network related.

I've had a random issue with my rendernodes falling out of the queue. Quite a few of those times, I'll try and retrieve the saved MXI file from the nodes' temp folders but Windows Explorer crashes the instant I even touch the parent folder of the MXI file. I think what's happening is that the rendernode is creating the MXI file, Windows Explorer crashes, and then waits for a response about restarting the application before anything else continues.

For kicks (and I don't recommend this), I even tried making a shortcut on my desktop directly to that temp directory (C:\<USER APP DATA LOCAL PATH>\Temp\mxnetwork\rendernode_<HOSTNAME>). This put Explorer in an endless crash loop. I had to delete the shortcut via command line that I launched from the task manager. Again, I don't recommend doing that

I uninstalled the Shell Extension and the crash stopped. I saw a similar post about MXM files, explorer, and the shell extension (http://www.maxwellrender.com/forum/view ... +extension), but this is a little different.

I'm using the following setup:
Win 7 Enterprise x64 (SP1)
Maxwell Render 3.0.1.3

Does the extension do more than generate thumbnails? Does it implement some graceful degradation in the event it runs into a problem, like pulling a default image for the thumbnail. I'm a bit hesitant to remove the shell extension throughout my network. What if there really is a corruption in the MXI files? It could potentially continue generating corrupted MXI's if the shell extension (or something else) doesn't validate the MXI file.

The zipped MXI file can be downloaded here:

http://csimages.c-sgroup.com/external_f ... plorer.zip

I can't even open the MXI file in Maxwell Render, so I'm assuming that it's in fact corrupt. Is it possible that the shell extension corrupted the MXI file, or is it much more likely that the file was corrupt initially, causing the shell extension to crash? I put it in a ZIP archive to prevent any crashes for people who download it. Fair warning once you unzip it and browse it; if you have work or things that you want to remain open in other Windows Explorer sessions, be sure to finish those so you don't lose where you were at.

Here's what the rendernode's log shows for the last received job. Notice how it just stops after writing the MXI file. Below these log entries is some of the log from the manager as well, which continues to communicate with the rendernode even after it crashed.

------------------------------------------------------------------------------------
Start
------------------------------------------------------------------------------------
maxwell:[INFO]: Installing error mode handler
maxwell:[INFO]:
maxwell:[INFO]:
maxwell:[INFO]: MAXWELL RENDER (M~R). Engine version 3.0.1.3
maxwell:[INFO]: (C) 2004-2013. Licensed to Next Limit Technologies
maxwell:[INFO]:
maxwell:[INFO]: C:\Program Files\Next Limit\Maxwell 3\maxwell.exe -nowait -mxs:Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\animations\DGTR-400_Rev00_0050.mxs -o:C:\Users\mktgdev\AppData\Local\Temp\mxnetwork\rendernode_mktgCGI-RN03\mxo0000050.png -mxi:C:\Users\mktgdev\AppData\Local\Temp\mxnetwork\rendernode_mktgCGI-RN03\DGTR-400_Rev00_0050.mxi -noimage -s:15 -t:999 -mintime:0 -node -nodeport:45463 -renameoutput -idcpu:3290 -slupdate:10 -v:4 -depth:8 -res:1920x1080 -camera:Camera.Animated -motionblur:off -displacement:off -dispersion:off -noimage:on -nomxi:off -ml:off -channel:alpha,off,16,png -channel:shadow,off,16,png -channel:object,off,16,png -channel:material,off,16,png -channel:motion,off,16,png -channel:zbuffer,off,16,png -channel:roughness,off,16,png -channel:fresnel,off,16,png -channel:normals,off,16,png -channel:position,off,16,png -channel:deep,off,32,exr -channel:uv,off,16,png -channel:alpha_custom,off,16,png -normalsspace:world -positionspace:world -zmin:0 -zmax:1 -deeptype:alpha -deepmindistance:0.2 -deepmaxsamples:20 -nomxi:off -rs:1
maxwell:[INFO]:
maxwell:[INFO]:
manually removed license info
maxwell:[INFO]:
maxwell:[INFO]:
maxwell:[INFO]: SYSTEM INFO
maxwell:[INFO]: Num processors: ( logical: 8, physical: 8 )
maxwell:[INFO]: Maxwell Root folder :C:\Program Files\Next Limit\Maxwell 3\
maxwell:[INFO]: Host ID: 082e5f22c7a0
maxwell:[INFO]: Low Priority Enabled
maxwell:[INFO]:
maxwell:[INFO]:
maxwell:[INFO]: Found 5 procedural extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
maxwell:[INFO]: MaxwellHair
maxwell:[INFO]: MaxwellParticles
maxwell:[INFO]: MaxwellVolumetric
maxwell:[INFO]: MGrassH
maxwell:[INFO]: MGrassP
maxwell:[INFO]: Found 5 geometry loader extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
maxwell:[INFO]: MaxwellMesher
maxwell:[INFO]: MaxwellSea
maxwell:[INFO]: MWObjectAlembic
maxwell:[INFO]: RFMeshes
maxwell:[INFO]: RWMeshes
maxwell:[INFO]: Found 4 geometry modifier extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
maxwell:[INFO]: MaxwellCloner
maxwell:[INFO]: MaxwellGrass
maxwell:[INFO]: MaxwellScatter
maxwell:[INFO]: SubdivisionModifier
maxwell:[INFO]: Found 10 texture extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
maxwell:[INFO]: Brick
maxwell:[INFO]: Checker
maxwell:[INFO]: Circle
maxwell:[INFO]: Gradient3
maxwell:[INFO]: Gradient
maxwell:[INFO]: Grid
maxwell:[INFO]: Marble
maxwell:[INFO]: Noise
maxwell:[INFO]: Voronoi
maxwell:[INFO]: WireframeTexture
maxwell:[INFO]: Found 7 material modifier extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
maxwell:[INFO]: AGS
maxwell:[INFO]: Opaque
maxwell:[INFO]: Transparent
maxwell:[INFO]: Metal
maxwell:[INFO]: Translucent
maxwell:[INFO]: Car Paint
maxwell:[INFO]: Hair
maxwell:[INFO]:
maxwell:[INFO]:
maxwell:[INFO]: Connecting to render node...
maxwell:[11/September/2014 06:01:11] [INFO]: Message to render node: engine_version 3.0.1.3
maxwell:[INFO]: Connected to render node.
maxwell:[INFO]: Socket descriptor: 816
maxwell:[INFO]:
maxwell:[11/September/2014 06:01:11] [INFO]: Reading MXS:Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\animations\DGTR-400_Rev00_0050.mxs
maxwell:[11/September/2014 06:01:12] [INFO]: MXS file read successfully
maxwell:[INFO]:
maxwell:[INFO]: CPU ID set by command line: 3290
maxwell:[INFO]: CPU ID used: 3290
maxwell:[INFO]:
maxwell:[11/September/2014 06:01:12] [INFO]: Checking scene dependencies..
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\HDRI Environment maps\Lightsmith\SoftBox_A\SoftBox_A\SoftBox_A.exr
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Steel-Dark-Mill_Extruded\SS06_Metal_5_B_adjusted.png
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Steel-Dark-Mill_Extruded\Steel-Dark-Mill_extruded_RWS_to_4_inches_Rev00.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Custom_Textures\Maxwell_Editor_Exports\textures\Metal_brushes_mix_map_normalized.png
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Steel-Dark-Polished_brushed\Steel-Dark-Polished_brushed_RWS_to_4_inch_Rev00.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Concrete\textures\Concrete_Normal.png
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Concrete\textures\Concrete_Diffuse.png
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\sceneassets\images\textures\Asphalt_01bump.JPG
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\sceneassets\images\textures\Asphalt_01.JPG
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\materials database\mxm files\Flooring\Carpet_Padding_Rev00.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Aluminum-Brushed_extruded\Aluminum-Brushed_extruded_RWS_to_1_inch_Rev00.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Aluminum-Polished_brushed\Aluminum-Polished_brushed_RWS_to_1_inch_Rev00.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Plastics\Dual_Durometer\white_plastic_Rev03.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Plastics\Dual_Durometer\EJC_Gasket_Greige_Rev09.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Maxwell Docs\Scene and Material Calibration\Plastics\Dual_Durometer\EJC_Gasket_Greige_Non-Textured_Rev02.mxm
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\MXSRefs\Carpet.Block.Non-Centered.mxs
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\sceneassets\images\textures\Carpet_Bristle_03.jpg
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\MXSRefs\turnbar.zeroed.mxs
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\MXSRefs\Static.Geometry.mxs
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\CS Products\EJC\All_Products-Max2013\export\Floors\DGTR-400\MXSRefs\Carpet.Block.Centered.mxs
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\HDRI Environment maps\Dosch\Studio Atmospheres\DH-Studio-Atmospheres_Spherical\StudioAtm_07_xxl_02px_blurred.hdr
maxwell:[11/September/2014 06:01:14] [INFO]: Dependency found: Y:\Custom_Textures\Light_Cutting_Masks\Studio_Reflection_Light_box_stacked_with_Hue.hdr
maxwell:[11/September/2014 06:01:14] [INFO]: Message to render node: render_info preprocessing data...
maxwell:[11/September/2014 06:01:14] [INFO]: Checking Data
maxwell:[11/September/2014 06:01:14] [INFO]: Loading Bitmaps & Preprocessing Data
maxwell:[11/September/2014 06:01:14] [DEBUG]: Preprocessing data.
maxwell:[11/September/2014 06:01:14] [DEBUG]: Preprocessing scene modifiers.
maxwell:[11/September/2014 06:01:14] [DEBUG]: Loading geometry extensions.
maxwell:[11/September/2014 06:01:14] [DEBUG]: Preprocessing MXS references.
maxwell:[11/September/2014 06:01:14] [INFO]: Loading MXS references...
maxwell:[11/September/2014 06:01:15] [INFO]: MXS references loaded successfully
maxwell:[11/September/2014 06:01:15] [DEBUG]: Pretessellating meshes with displacement.
maxwell:[11/September/2014 06:01:15] [DEBUG]: Initializing data.
maxwell:[11/September/2014 06:01:15] [DEBUG]: Preprocessing geometry.
maxwell:[11/September/2014 06:01:15] [DEBUG]: Preprocessing materials.
maxwell:[11/September/2014 06:01:27] [DEBUG]: Preprocessing additional parameters.
maxwell:[11/September/2014 06:01:27] [DEBUG]: Initializing render engine.
maxwell:[11/September/2014 06:01:27] [WARNING]: DO NOT SAVE IMAGE FILE flag enabled. Image file will not be saved
maxwell:[11/September/2014 06:01:27] [DEBUG]: Initializing multilight data.
maxwell:[11/September/2014 06:01:27] [INFO]: Starting voxelization.
maxwell:[INFO]:
maxwell:[INFO]: Scene Info:
maxwell:[INFO]: Image output: C:\Users\mktgdev\AppData\Local\Temp\mxnetwork\rendernode_mktgCGI-RN03\mxo0000050.png
maxwell:[INFO]: MXI output: C:\Users\mktgdev\AppData\Local\Temp\mxnetwork\rendernode_mktgCGI-RN03\DGTR-400_Rev00_0050.mxi
maxwell:[INFO]:
maxwell:[INFO]: Geometry:
maxwell:[INFO]: - Num Objects: 486
maxwell:[INFO]: - Num Meshes: 35
maxwell:[INFO]: - Num Triangles: 3080650
maxwell:[INFO]: - Num Vertexes: 1567062
maxwell:[INFO]: - Num Normals: 1432411
maxwell:[INFO]: - Num Materials: 22
maxwell:[INFO]:
maxwell:[INFO]: Camera: Camera.Animated
maxwell:[INFO]:
maxwell:[INFO]: Render settings:
maxwell:[INFO]:
maxwell:[INFO]: - Engine version : 3.0.1.3
maxwell:[INFO]: - Using RS1 engine
maxwell:[INFO]: - Desired rendering time : 999
maxwell:[INFO]: - Desired sampling level : 15.000000
maxwell:[INFO]: - Using 8 threads
maxwell:[INFO]: - Multilight type: Disabled
maxwell:[INFO]: - Save lights in separate files:No
maxwell:[INFO]: - Motion blur: disabled
maxwell:[INFO]: - Displacement: disabled
maxwell:[INFO]: - Dispersion: disabled
maxwell:[INFO]: - Illumination layers:
maxwell:[INFO]: . . direct layer: true
maxwell:[INFO]: . . indirect layer: true
maxwell:[INFO]: . . direct caustic reflection layer: true
maxwell:[INFO]: . . direct caustic refraction layer: true
maxwell:[INFO]: . . indirect caustic reflection layer: true
maxwell:[INFO]: . . indirect caustic refraction layer: true
maxwell:[INFO]:
maxwell:[11/September/2014 06:01:27] [INFO]: Start Voxelization
maxwell:[11/September/2014 06:01:27] [INFO]: Message to render node: render_info starting voxelization...
maxwell:[11/September/2014 06:01:39] [INFO]: End Voxelization
maxwell:[11/September/2014 06:01:39] [INFO]: Message to render node: render_info voxelization done.
maxwell:[11/September/2014 06:01:39] [INFO]: Voxelization done.
maxwell:[11/September/2014 06:01:39] [INFO]: Start Rendering
maxwell:[INFO]:
maxwell:[11/September/2014 06:01:39] [INFO]: Message to render node: render_started
maxwell:[INFO]: Start writing MXI file...
maxwell:[11/September/2014 06:01:45] [INFO]: Message to render node: start_writing_mxi
maxwell:[INFO]: End writing MXI file...
maxwell:[11/September/2014 06:01:45] [INFO]: Message to render node: end_writing_mxi
maxwell:[11/September/2014 06:01:45] [INFO]: MXI successfully renamed.
maxwell:[11/September/2014 06:01:46] [INFO]: Benchmark of 319.015. Time: 6s. SL of 1.00
maxwell:[11/September/2014 06:01:46] [INFO]: Message to render node: time_update 6
maxwell:[11/September/2014 06:01:46] [INFO]: Message to render node: new_sampling_level_reached 1.00000
maxwell:[11/September/2014 06:01:46] [INFO]: Time left: 1h15m22s
maxwell:[11/September/2014 06:01:49] [INFO]: Message to render node: time_update 9
maxwell:[11/September/2014 06:01:49] [INFO]: Message to render node: new_sampling_level_reached 1.296486
maxwell:[INFO]: Start writing MXI file...
maxwell:[11/September/2014 06:01:55] [INFO]: Message to render node: start_writing_mxi
maxwell:[INFO]: End writing MXI file...
maxwell:[11/September/2014 06:01:56] [INFO]: Message to render node: end_writing_mxi
maxwell:[11/September/2014 06:01:56] [INFO]: MXI successfully renamed.
maxwell:[11/September/2014 06:01:57] [INFO]: Benchmark of 296.256. Time: 17s. SL of 2.00
maxwell:[11/September/2014 06:01:57] [INFO]: Message to render node: time_update 17
maxwell:[11/September/2014 06:01:57] [INFO]: Message to render node: new_sampling_level_reached 2.00013
maxwell:[11/September/2014 06:01:57] [INFO]: Time left: 1h34m10s
maxwell:[11/September/2014 06:01:59] [INFO]: Message to render node: time_update 19
maxwell:[11/September/2014 06:01:59] [INFO]: Message to render node: new_sampling_level_reached 2.161549
maxwell:[11/September/2014 06:02:09] [INFO]: Message to render node: time_update 29
maxwell:[11/September/2014 06:02:09] [INFO]: Message to render node: new_sampling_level_reached 2.840874
maxwell:[INFO]: Start writing MXI file...
maxwell:[11/September/2014 06:02:11] [INFO]: Message to render node: start_writing_mxi
maxwell:[INFO]: End writing MXI file...
maxwell:[11/September/2014 06:02:12] [INFO]: Message to render node: end_writing_mxi
maxwell:[11/September/2014 06:02:12] [INFO]: MXI successfully renamed.
maxwell:[11/September/2014 06:02:13] [INFO]: Benchmark of 294.021. Time: 33s. SL of 3.00
maxwell:[11/September/2014 06:02:13] [INFO]: Message to render node: time_update 33
maxwell:[11/September/2014 06:02:13] [INFO]: Message to render node: new_sampling_level_reached 3.00002
maxwell:[11/September/2014 06:02:13] [INFO]: Time left: 1h34m27s
maxwell:[11/September/2014 06:02:19] [INFO]: Message to render node: time_update 39
maxwell:[11/September/2014 06:02:19] [INFO]: Message to render node: new_sampling_level_reached 3.309474
maxwell:[11/September/2014 06:02:29] [INFO]: Message to render node: time_update 49
maxwell:[11/September/2014 06:02:29] [INFO]: Message to render node: new_sampling_level_reached 3.755139
maxwell:[INFO]: Start writing MXI file...
maxwell:[11/September/2014 06:02:35] [INFO]: Message to render node: start_writing_mxi
maxwell:[INFO]: End writing MXI file...
maxwell:[11/September/2014 06:02:36] [INFO]: Message to render node: end_writing_mxi
------------------------------------------------------------------------------------
RENDERNODE END
------------------------------------------------------------------------------------



------------------------------------------------------------------------------------
MANAGER START
------------------------------------------------------------------------------------
[11/September/2014 06:03:14] Node: mktgCGI-RN03: time_update: 49
[11/September/2014 06:03:14] Node: mktgCGI-RN03: new_sampling_level_reached: 3.755139
[11/September/2014 06:03:14] Render node: mktgCGI-RN03 Job ID: 19 frame:50 sl: 3.75514
~miscellaneous log entries~
[11/September/2014 06:03:21] Node: mktgCGI-RN03: time_update: 57
[11/September/2014 06:03:21] Node: mktgCGI-RN03: new_sampling_level_reached: 4.00094
[11/September/2014 06:03:21] Render node: mktgCGI-RN03 Job ID: 19 frame:50 sl: 4.00094
~miscellaneous log entries~
[11/September/2014 06:03:24] Node: mktgCGI-RN03: time_update: 60
[11/September/2014 06:03:24] Node: mktgCGI-RN03: new_sampling_level_reached: 4.073177
[11/September/2014 06:03:24] Render node: mktgCGI-RN03 Job ID: 19 frame:50 sl: 4.07318
~miscellaneous log entries~
[11/September/2014 06:06:14] Node: mktgCGI-RN03: time_update: 230
[11/September/2014 06:06:14] Node: mktgCGI-RN03: new_sampling_level_reached: 7.077986
[11/September/2014 06:06:14] Render node: mktgCGI-RN03 Job ID: 19 frame:50 sl: 7.07799
~miscellaneous log entries~
[11/September/2014 06:07:37] Connection with render node enabled.
[11/September/2014 06:07:37] ERROR: Render node already registered in the manager.
[11/September/2014 06:07:37] ERROR: Error adding render node to list
[11/September/2014 06:07:37] Node: mktgCGI-RN03: node_local_name: mktgCGI-RN03
[11/September/2014 06:07:37] Node: mktgCGI-RN03: node_os: Windows 64
[11/September/2014 06:07:37] Node: mktgCGI-RN03: node_version: 3.0.1.3
[11/September/2014 06:07:37] Node: mktgCGI-RN03: stop_render_order_received_before_rendering:
[11/September/2014 06:07:37] Connection through alternative port enabled.
[11/September/2014 06:07:37] Add alternative socket for node: node
[11/September/2014 06:07:37] Socket descriptor: 3468
[11/September/2014 06:07:37] Socket local port: 45464
[11/September/2014 06:07:37] Socket peer port: 49162
[11/September/2014 06:07:37] Connection with render node enabled.
[11/September/2014 06:07:37] ERROR: Render node already registered in the manager.
[11/September/2014 06:07:37] ERROR: Error adding render node to list
~miscellaneous log entries~
[11/September/2014 06:07:39] ERROR: Error in render node: mktgCGI-RN03! Unable to write
------------------------------------------------------------------------------------
MANAGER END
------------------------------------------------------------------------------------

[JDHill]

The shell extension only reads the specified MXI file, and it uses regular SDK methods to do so (i.e. CmaxwellMxi::read(path)). If the read fails, the plugin simply pulls a bitmap (of the MXI icon) from its resources and hands that to Explorer. At no time does it attempt to alter the MXI file, or otherwise write anything to disk, implying that any corruption would have been written into the MXI file itself. That said, I confirm that the provided MXI does indeed crash Explorer, so thanks for providing that; I'll take a look and see how/why that is happening, and whether there's some way to avoid it (in other words, see if there's some way to handle corrupt MXI files without crashing).

As for how an MXI might get corrupted, I couldn't say. It could be written that way, or it could be due to another factor (e.g. disabled write-cache buffer flushing in the OS, etc).

[zparrish]

Thanks for the info JD! That's actually really reassuring that the shell can't corrupt the MXI files. I'll reinstall it on that node.

I'll see if I can find any likely candidates for file corruption within the OS and hardware. The really weird part is that two other nodes crashed at the same time, but their files weren't corrupted. I don't know if it all relates to the networking tools in Maxwell or if it was some other fluke. Regardless, the likely hood that this is related to the core Maxwell Render engine is unlikely, and this post can probably be moved to the "Maxwell Network" forum (http://www.maxwellrender.com/forum/viewforum.php?f=72)

I also didn't document prior incidents, so it's possible that previous Windows Explorer crashes and corrupt MXI files were happening on the same node, indicating an issue local to that machine.

Would it be possible for one node to write a corrupt MXI (regardless of the reason), and then some erratic behavior between that node and the manager causes other nodes to crash?

Thanks again JD!

[zparrish]

I ran some hardware diagnostics (Spinrite and Memtest) and that all checked out. I also checked the Write-caching policies on these nodes and they were set to the default for Windows 7, which is enabling the "Enable write caching on this device" option, but not enabling the "Turn off Windows write-cache buffer flushing on the device" option.

To improve stability, I completely disabled write-caching on all these nodes, primarily because none of them are hooked up to a UPS. I would also like to add that the facility's industrial 3-phase power utility feed is spread out internally with a system called DELTA for load balancing. I've tested the voltage here before and it's kind of scary. My residential single phase feed has about a +/- 1 volt, where as I've seen it fluctuate nearly 6 volts here in the facility. I guess for now, I'll chalk it up to a power issue (or a glitch in the Matrix ) and move on. A power flash would also explain why two other nodes lost connectivity to the manager at the same time.

Sorry to stray off topic. Thanks again JD!

[JDHill]

I've run through opening this file in the debugger, and the most I can really tell you at this point is that it is simply corrupt. I cannot speculate on the process by which it might've come to be so, but I'll talk to the relevant developers, and see what we can do to handle such situations more gracefully.