I'll try to reply to everyone at once:
Laurier; you cannot prevent leaf litter from rotting, and I wouldn't want to. So you're not having problems, your tank is doing what it's supposed to do, in my opinion.
A biologically active tank will eat leaf litter in time, just add more. I use oak leaves that over time decompose due to the action of bacteria, micro-fauna and fungi...green algae is often the first thing I spot on them.
Nanashi, Rachel; it doesn't usually take long, but it does take practice. I select the plants with an eye to locality and authenticity (more on that later...sometimes I make compromises, sometimes I don't, sometimes I make a guess. Sometimes I'm horribly wrong!)
I then "fake" nature. What I mean by that, is plants in their natural habitat do what they have evolved to do, without regard for what humans find pleasing. This is why if you walk into a forest and start taking random photos, the results will likely be disappointing, because most of what you get on film (or CCD if you go digital) will be a tangle of lines that are meaningless to human eyes, lacking a discernible pattern or reference point.
As humans we look for patterns, proportion, pleasing lines, and visual context, it's what our brains have evolved to do, and have become adapted to doing in all situations, i.e. what began in a distant, arboreal ancestor as a need to find horizontal lines in the treetops to aid climbing, later in yet another ancestor to aid in mate selection, predator recognition, distance estimation to aid in the hunt, etc. etc., has resulted in certain things jumping out in our visual field and yelling "Hey! Look at me!"
Or at least that's my story and I'm sticking to it. This need for eye candy carries over into design. While it's possible to throw a bunch of plants and branches together randomly and make them look awesome, it's best not to leave it to chance.
When designing a vivarium, I take into account what plants I want to use, what I'm trying to mimic, and then I do this:
1. Find a reference point. This could be a plant or a piece of wood, whatever. But this is where the eye gets drawn to first, or at least that's the idea.
2. Create a visual path through the space. Map out smaller reference points, usually from left to right, at different levels in the tank, give people something to look at as their eyes wander. This is usually shaped like some kind of triangle. Diagonal lines are good.
3. Use foresight; the plants will grow and change, they will require pruning, wood will rot, etc.. Nothing in a tank is static. The leaf litter rots.
4. Use "negative space"...spots in the tank which are framed by objects or plants, but are essentially open space. This can be a very dramatic and pleasing effect.
I mentioned earlier that the 90 gallon tank pictured is an early attempt; I don't design them quite like that anymore. You'll notice many species of plant in there. While it looks nice, to my mind it's more of a garden than a mimic of nature...consider that even in a rainforest with tons of biodiversity, you won't usually (this from lots of research and sifting through firsthand accounts) find every species crammed into a small space. A patch of one particular plant may continue for several yards. Epiphytic plants are usually high up and difficult to see.
This topic was brought up by Brent Brock (a very knowledgeable guy who has me totally outclassed, one of the people I bug for plant and frog knowledge) in an essay he wrote, where he pointed out that lots of viv design is "fractal" .. that is, a compressed version of a much larger pattern. I wanted to get away from this style of design, and opted instead for simplicity. To my eyes it's more realistic.
This is not to say that in a rainforest habitat, there aren't riverbanks and breaks in the canopy that encourage a dense explosion of undergrowth, but I'm after something very specific...something closer in scale to a specific scene one would find in nature, and at the same time, tweaked for human eyes. This is where Mr. Realistic Habitat starts cheating like mad.
Remember waaaay back in the beginning of this stupidly long post, I said that plants do whatever they want and don't care if it looks pretty? So I intervene by trying to create visual balance through use of different plant types, sizes, and growth habits. Maybe I'll take a broad leaf species and set a smaller, lower, ground hugging species underneath and in front of it. This is a basic tactic for formal gardens, but by skewing the position of things off-centre, or creating breaks in the ground cover, I can maybe provide the illusion that it's "natural".
Maybe within a couple of months one of my vines has exploded into lush growth and is filling the tank. This is when the heavy pruning starts, because I want the vine I planted, not the one that's grown. If you're careful about pruning (even if you're not, with some species) the plant will recover, and your original design, although changed, will remain in some form or other.
In my increasing trend towards simplicity and my careful management of composition and shape, I'm getting closer to a Japanese aesthetic principle...a reverence for nature that results in an idealized, very human imitation, that's both very elegant and simple, and under a great deal of control. Amano style natural aquariums (GORGEOUS) are a great example of this. So are Bonsai trees, not the cheap imitations you find everywhere, the real ones with many years invested in them by masters.
I haven't even come close to succeeding yet and I probably never will, although along the way I'm getting closer to it, and modifying the practical aspects of vivarium design into something that works for me, since there are many schools of thought regarding planting substrate, drainage, filtration, etc..
More on these practical aspects another time, since you guys are probably asleep by now.
I'll also write a separate post on plant species I've used, lighting, water, growth, etc.. Although after this snore-fest you'll probably never read my posts again!
And, I suppose I should post photos of a couple of my "transition" tanks...the simpler design I'm pursuing but haven't got down yet.
By the way, Chris, how's that vivarium I made for you? Have you found those diagonal, curving branches to add to it yet?