This document describes some of the important issues with implementing grids, and suggests solutions for them.
FRAMESET tags can certainly be nested, as per the Frame Syntax description. But such a structure is basically flattened before display on the screen. A more interesting issue is: can a FRAME contain a document which is itself a FRAMESET? The short answer is a resounding yes. Of course FRAMESETS must be nestable, especially with named Frames so that a link in one frame can reload multiple other frames.
It is possible to design a FRAMESET that will recurse infinitely stacking the same FRAMESET inside itself until the machine ran out of memory. Consider the following example:
< FRAME src=parents_url>
This is obviously bad, so it has been prevented. Any frame that attempts to assign its SRC URL to be the same as the URL of any of its ancestors will be treated as if it has no SRC URL at all (basically a blank frame). While this doesn't stop all malicious documents, it eliminates a whole troublesome class of them.