sorry, I didn't read well your post Daedalus the first time.
So you are talking about non-maxwell renders... hmmm mental ray? What I am going to say applies to mental ray and Maxwell.
Hmmm how about this,
Open your maya scene file, let us say it is scene 1.
From within this scene create a reference to tree.mb (the scene that contains the tree). This will show a scene with scene 1 geometry and tree.mb geometry. However, scene 1 will only be as large as it was before (no impact in disk space while working on it) but will render scene1 and tree. (ok it will increase in size by a hundred kb or so, but it is negligible)
I guess Daedalus that your question really is what is more efficient at rendering time, instancing inside the scene, instancing a reference or referencing an instance. I guess what you are trying to do is separate the file in parts that are manageable to work on but renderable in the most powerful machine you have available. It seems to me that it wouldn't make a noticeable difference any method in a 64-bit system that can render all three cases, but maybe someone has a more informed opinion?
There are several things you could try. First is instancing the referenced object inside the scene file. Another is instancing in the reference scene. And yet a third one is saving everything in the same scene and then instancing there, I'll tell you down below why I suggest this.
If you are going for a forest (say thousands of these trees) one way it would seem to be to make a forest.mb scene with one tree and 999 instanced ones. Then reference this from the scene1.mb file. Seems to me this would be easier to manage.
Placement of the trees would be a little tricky, but if you designate a low-res proxy I guess it would be easier to see in the scene.
The problem I have had in Maxwell is that I know for a fact that it sometimes messes up instanced objects placement, specially if you change the scale of the tree (as I guess you would want to do in a forest). This has happenned to me with some complex instances (say you have a group, and then you parent it and then you instance it and then you parent it and then it is complex enough that it doesn't render properly)
In some of the files that I have worked with, the mxcl renderer definitely messed up some instances, so what I ended up doing is converting them to regular geometry right before rendering, otherwise it would not render correctly. Since this was unavoidable I ended up using layers and hiding them while working. Then for render I select all instances that were giving me problems and then convert them and render.
I guess it will be fixed at some point, but since you mentioned only 4 or 5 trees it seems you are better off just loading a reference and instancing it in the scene that you are working on. But if you can render this and hard disk space is of no concern, why not put everything in the maya file? then it seems that you would only hit a limit if you are working on a 32-bit renderer. If this is the case, then switching to a 64-bit renderer (Maxwell or otherwise) would be better for you than playing around with instances.
Then I don't think you would have a problem rendering 5 or 10 trees with 2 million polys each plus whatever scene you have. (Assuming of course you have RAM to render this)
I digressed a little but I wanted to suggest other approaches if they are helpful to you. Maybe instancing is not that helpful with just 5 2 million poly-trees?
I am curious to see what other people have to say.