[Daniel Kerbler 20190926093923]

Hi,

just trying out the GPU renderer and the denoise functionalities.

One thing that is striking me, when I observed the console is the super frequent "merging" and image/mxi writing operations, that appear to leave the gpus unused for a significant amount of time. I tried to supress with -mintime: but it does not seem to have an effect, judging by what I can see in the console.
Looks like there is room for imporvement, or is there something I can do about this on my side?

best,
Daniel

console:


Gpu usage patterns logged by GPU-Z of the two cards (not concurrent)

[luis.hijarrubia]

You are right on pointing the room for improvement. You can't do anything about this on user side.
The render takes more time before updating on high SL.
It a case of balance. On low sl we want to show results quick, that's why there so many updates. If we set updates to be less frequent somebody with a low-medium graphic card should have to wait a lot to have updates. If we put them very frequently a high-end card looses a lot of computing time at the beggining of the render.
We will re-visit this balance soon, as I think a less frequently update pattern would be good with current cards.

[Daniel Kerbler 20190926093923]

thanks for clarifying Luis.

so it is a multi-GPU related issue.

When doing a multi-node cpu rendering(aka networkrender) we can use a button to preview the merged result.

wouldn't that be an option for multi-GPU too?
> if it is not possible to display the cooperative GPU image on the fly, only show the render of one card(the fastest if different) and then have a "preview" button for merging the separate GPU MXIs.

It would be really helpful to have some user control over defining the speed of the update pattern.
sometimes one needs to see the render in the beginning (test renderings ) and in other scenarios it is not important at all. (often high SL is needed to evaluate quality correctly)
I think it would be hard for you on the coding side to guess for these scenarios.

[Brany]

Maybe there is something you can do. Running maxwell.exe with command-line flag "-mintime:n" where "n" means how many minutes you want to force to wait between save-to-disk events, you can set, for example, 10 minutes, and maximize the render time.

[luis.hijarrubia]

thanks for clarifying Luis.

so it is a multi-GPU related issue.

When doing a multi-node cpu rendering(aka networkrender) we can use a button to preview the merged result.

wouldn't that be an option for multi-GPU too?
> if it is not possible to display the cooperative GPU image on the fly, only show the render of one card(the fastest if different) and then have a "preview" button for merging the separate GPU MXIs.

It would be really helpful to have some user control over defining the speed of the update pattern.
sometimes one needs to see the render in the beginning (test renderings ) and in other scenarios it is not important at all. (often high SL is needed to evaluate quality correctly)
I think it would be hard for you on the coding side to guess for these scenarios.

No, that's not only on multi-gpu, on single gpu happens too. You need to copy samples from GPU to CPU, and GPU may add new samples before the copy is done. So we need to stop GPU until the copy is performed.

As for having a button or something similar, let's say <<Image on demand>>, it's not the way maxwell render have been working for 15 years. We don't want to have that behaviour difference between engines.

You can set update time with command line:

-slupdate:seconds Specify the time for refreshing the sampling level info
-mintime:minutes Specify the minimum time for saving to disk. Default: 0 (automatic). Set a very high value to save only at end

We will work on default values for gpu to improve performance, but we want to be carefull, as high times can be frustrating if render crashes and you don't have any temporal MXI or image.

[Daniel Kerbler 20190926093923]

If we put them very frequently a high-end card looses a lot of computing time at the beggining of the render.

this is not what I am observing. Actually the other way round. something is strange here!
Please see the log from the console below.
I set -mintime:60 and slupdate:1200 but I am not sure if it actually changers anything the cards still merge right away, but not so frequent at start, but after a while they are literally merging all the time!?



- Engine version : 5.0.0.14
- Using RSC engine
- Desired rendering time : 9999
- Desired sampling level : 22.000000
- Using 24 threads
- Multilight type: Disabled
- Save lights in separate files:No
- Motion blur: enabled
- Displacement: enabled
- Dispersion: enabled
- Denoiser: enabled
- Illumination layers:
. . direct layer: true
. . indirect layer: true
. . direct caustic reflection layer: true
. . direct caustic refraction layer: true
. . indirect caustic reflection layer: true
. . indirect caustic refraction layer: true

[26/November/2019 19:27:21] Start Voxelization
[26/November/2019 19:27:29] End Voxelization
[26/November/2019 19:27:29] Voxelization done.
[26/November/2019 19:27:29] Memory used after voxelization: 3751 Mb
[26/November/2019 19:28:14] Start Rendering

[26/November/2019 19:28:14] Rendering using 2 cards.
[26/November/2019 19:28:14] Card 1: GeForce GTX TITAN X.
[26/November/2019 19:28:14] Card 2: GeForce GTX TITAN X.
[26/November/2019 19:28:19] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:23] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:26] Benchmark of 766.518. Time: 50s. SL of 3.57
[26/November/2019 19:28:26] Time left: 1h54m42s
[26/November/2019 19:28:27] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:29] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:36] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:38] Benchmark of 2461.914. Time: 1m02s. SL of 6.49
[26/November/2019 19:28:38] Time left: 4h37m19s
[26/November/2019 19:28:41] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:43] Benchmark of 3129.238. Time: 1m07s. SL of 7.22
[26/November/2019 19:28:43] Time left: 8m00s
[26/November/2019 19:28:46] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:28:47] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:02] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:04] Benchmark of 3898.728. Time: 1m28s. SL of 8.38
[26/November/2019 19:29:04] Time left: 3h50m33s
[26/November/2019 19:29:07] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:09] Benchmark of 4712.091. Time: 1m33s. SL of 8.97
[26/November/2019 19:29:09] Time left: 7h18m50s
[26/November/2019 19:29:22] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:25] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:56] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:29:58] Benchmark of 5505.432. Time: 2m22s. SL of 10.36
[26/November/2019 19:29:58] Time left: 4h45m23s
[26/November/2019 19:30:01] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:30:04] Benchmark of 6441.041. Time: 2m28s. SL of 10.84
[26/November/2019 19:30:04] Time left: 8h16m11s
[26/November/2019 19:30:42] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:30:51] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:31:51] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:31:53] Benchmark of 6878.280. Time: 4m18s. SL of 12.36
[26/November/2019 19:31:53] Time left: 4h45m55s
[26/November/2019 19:32:00] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:32:02] Benchmark of 8033.944. Time: 4m26s. SL of 12.81
[26/November/2019 19:32:02] Time left: 3h17m33s
[26/November/2019 19:33:33] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:34:08] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:36:01] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:36:03] Benchmark of 7870.419. Time: 8m27s. SL of 14.34
[26/November/2019 19:36:03] Time left: 5h02m40s
[26/November/2019 19:36:51] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:36:54] Benchmark of 8554.185. Time: 9m18s. SL of 14.79
[26/November/2019 19:36:54] Time left: 4h52m49s
[26/November/2019 19:39:48] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:40:51] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:04] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:07] Benchmark of 9347.963. Time: 13m31s. SL of 15.92
[26/November/2019 19:41:07] Time left: 50m37s
[26/November/2019 19:41:32] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:40] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:42] Benchmark of 9164.570. Time: 14m06s. SL of 15.98
[26/November/2019 19:41:42] Time left: 1h54m57s
[26/November/2019 19:41:46] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:50] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:41:52] Benchmark of 9102.181. Time: 14m16s. SL of 15.99
[26/November/2019 19:41:52] Time left: 1m53s
[26/November/2019 19:41:56] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:00] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:03] Benchmark of 9030.814. Time: 14m27s. SL of 16.00
[26/November/2019 19:42:03] Time left: 1h57m25s
[26/November/2019 19:42:06] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:10] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:13] Benchmark of 8971.443. Time: 14m37s. SL of 16.01
[26/November/2019 19:42:13] Time left: 1h57m56s
[26/November/2019 19:42:15] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:17] Benchmark of 9407.526. Time: 14m41s. SL of 16.14
[26/November/2019 19:42:20] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:24] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:27] Benchmark of 9366.287. Time: 14m51s. SL of 16.16
[26/November/2019 19:42:27] Time left: 2s
[26/November/2019 19:42:30] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:34] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:37] Benchmark of 9304.775. Time: 15m01s. SL of 16.17
[26/November/2019 19:42:37] Time left: 40m38s
[26/November/2019 19:42:40] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:43] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:44] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:47] Benchmark of 9391.319. Time: 15m11s. SL of 16.22
[26/November/2019 19:42:47] Time left: 43m25s
[26/November/2019 19:42:50] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:54] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:42:57] Benchmark of 9372.334. Time: 15m21s. SL of 16.24
[26/November/2019 19:42:58] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:00] Benchmark of 9373.110. Time: 15m25s. SL of 16.25
[26/November/2019 19:43:04] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:07] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:12] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:14] Benchmark of 9345.061. Time: 15m38s. SL of 16.28
[26/November/2019 19:43:14] Time left: 1m18s
[26/November/2019 19:43:16] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:18] Benchmark of 9366.207. Time: 15m42s. SL of 16.30
[26/November/2019 19:43:21] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:25] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:29] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:31] Benchmark of 9348.954. Time: 15m56s. SL of 16.33
[26/November/2019 19:43:33] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:36] Benchmark of 9349.802. Time: 16m00s. SL of 16.34
[26/November/2019 19:43:38] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:39] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:43] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:45] Benchmark of 9341.834. Time: 16m09s. SL of 16.36
[26/November/2019 19:43:45] Time left: 2m24s
[26/November/2019 19:43:47] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:50] Benchmark of 9333.108. Time: 16m14s. SL of 16.37
[26/November/2019 19:43:52] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:53] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:57] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:43:59] Benchmark of 9325.393. Time: 16m23s. SL of 16.39
[26/November/2019 19:43:59] Time left: 1m27s
[26/November/2019 19:44:01] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:04] Benchmark of 9316.877. Time: 16m28s. SL of 16.40
[26/November/2019 19:44:06] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:07] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:10] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:13] Benchmark of 9309.425. Time: 16m37s. SL of 16.42
[26/November/2019 19:44:13] Time left: 2s
[26/November/2019 19:44:15] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:17] Benchmark of 9301.105. Time: 16m42s. SL of 16.43
[26/November/2019 19:44:20] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:21] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:24] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:27] Benchmark of 9293.891. Time: 16m51s. SL of 16.45
[26/November/2019 19:44:27] Time left: 2s
[26/November/2019 19:44:29] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:31] Benchmark of 9294.910. Time: 16m55s. SL of 16.46
[26/November/2019 19:44:34] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:35] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:38] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:40] Benchmark of 9278.794. Time: 17m05s. SL of 16.48
[26/November/2019 19:44:40] Time left: 7m03s
[26/November/2019 19:44:43] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:45] Benchmark of 9279.860. Time: 17m09s. SL of 16.49
[26/November/2019 19:44:47] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:49] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:52] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:54] Benchmark of 9273.019. Time: 17m18s. SL of 16.51
[26/November/2019 19:44:54] Time left: 14m19s
[26/November/2019 19:44:56] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:44:59] Benchmark of 9265.199. Time: 17m23s. SL of 16.52
[26/November/2019 19:45:01] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:02] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:05] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:08] Benchmark of 9258.581. Time: 17m32s. SL of 16.54
[26/November/2019 19:45:08] Time left: 2m33s
[26/November/2019 19:45:10] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:13] Benchmark of 9250.938. Time: 17m37s. SL of 16.55
[26/November/2019 19:45:15] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:16] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:19] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:21] Benchmark of 9244.528. Time: 17m46s. SL of 16.57
[26/November/2019 19:45:21] Time left: 2s
[26/November/2019 19:45:24] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:27] Benchmark of 9237.043. Time: 17m51s. SL of 16.58
[26/November/2019 19:45:29] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:30] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:33] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:35] Benchmark of 9239.375. Time: 17m59s. SL of 16.60
[26/November/2019 19:45:35] Time left: 2s
[26/November/2019 19:45:38] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:41] Benchmark of 9223.509. Time: 18m05s. SL of 16.61
[26/November/2019 19:45:42] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:44] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:47] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:49] Benchmark of 9225.920. Time: 18m13s. SL of 16.63
[26/November/2019 19:45:49] Time left: 2s
[26/November/2019 19:45:52] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:55] Benchmark of 9210.322. Time: 18m19s. SL of 16.63
[26/November/2019 19:45:56] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:45:58] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:02] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:05] Benchmark of 9196.183. Time: 18m29s. SL of 16.65
[26/November/2019 19:46:05] Time left: 1m43s
[26/November/2019 19:46:08] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:10] Benchmark of 9206.349. Time: 18m34s. SL of 16.67
[26/November/2019 19:46:12] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:16] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:19] Benchmark of 9183.616. Time: 18m43s. SL of 16.68
[26/November/2019 19:46:19] Time left: 2s
[26/November/2019 19:46:21] Card 1: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:22] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:27] Card 2: GeForce GTX TITAN X is merging.
[26/November/2019 19:46:29] Benchmark of 9186.869. Time: 18m53s. SL of 16.70
[26/November/2019 19:46:29] Time left: 11m26s
[26/November/2019 19:46:32] Card 1: GeForce GTX TITAN X is merging.

[luis.hijarrubia]

It seems when it reaches SL 15 the pattern changes. My first impression is that a process we have, called thread mapping is messing there. We were going to re check all this process anyway.

[Daniel Kerbler 20190926093923]

alright, thanks for having a look
wanted to point that out, since it seemed strange to me