As has been discussed, what we call X, is actually a convergence of various components: X server, Window Manager, Desktop, etc. With MS Windows, the GUI desktop is tightly integrated with the operating system itself. This is not the case in Linux which follows the Unix tradition of combining various independent components to achieve some end result. So we have choices with each component and it's attendant configuration and implementation. In short, much flexibility. This is where you come in. You can try various possibilities and decide what you gives you the most bang for the buck.
On low end hardware, this gives us much latitude to decrease the demand on available system resources. This is good because, if given the opportunity, X can be quite greedy with system resources. If you've recently installed a new Linux distribution, you've probably been given a default Desktop with many bells and whistles. And something that will probably need a fair amount of memory and CPU to achieve a reasonable level of performance. If you have the horse power, this should not be a problem.
It is often said that Linux functions very well with relatively little memory. This is true to a point. It does not mean though that every possible configuration will run with low memory. So if you want to use memory hungry applications, then you will have to have the memory. Or you will have to make sacrifices to achieve a satisfactory level of performance. It is quite possible to run X with reasonable performance on 16 Meg of RAM, and even less if you really want to push it. But you would have to live with some real limitations.
Let's look at some of the components and ways to decrease the demand on system resources, in case you are at the low end on hardware, or performance is not up to expectations.
No big surprise, but overall system performance will be best with a fast graphics card, a fast hard drive, and lots and lots of memory -- if you want both a fast and flashy system.
Graphics cards are of course necessary, and the X server's video performance is tied to the card's chipset, and the corresponding XFree86 driver. Just because a given card is supported by XFree86 does not necessarily mean it is as well optimized as other cards! It may also perform better at a lower color depth (see below). It may well be worth the trip to xfree86.org to see if there are any notes related to your card with respect to performance or other issues.
And you might try other versions of XFree86. At this time v4.2 was just recently released. Some cards may still perform better with 3.3.6 due to the way v4.x is being incrementally developed. If you are using x4.x and performance is not good, then make sure you are using the latest available version.
The more memory, the better. X will do a lot caching to help performance. But caching requires memory, and if there isn't much to start with, then we would need to reduce memory requirements. Some tips for those with low memory or performance problems:
Use the free command to make sure all memory and swap is recognized.
Make sure you don't have other system services that are hogging memory or CPU. Use top or ps to see what is running, and disable anything you can to free up memory and CPU. Again, a default installation may have many things running that you don't really need.
Make sure you have plenty of swap space. With low, or even modest, memory, swap is all the more important. A general rule of thumb is twice as much swap as physical memory. With low memory, this is not enough. Try four times real memory. Or more. If you can't create more swap partitions, see the mkswap man page about creating swap files instead. Constant disk churning is a symptom of insufficient swap space, and the system will be slowed as a result, sometimes drastically. Or, possibly this may be the symptom of a poorly behaved kernel VM system (try another kernel in this case).
Drive performance is important for swap performance. Make sure your drive has DMA enabled if the drive supports it, and is otherwise tuned and performing up to snuff. See the hdparm man page. Slow drive + slow card + low memory = slow system.
Don't use KDE or GNOME if memory is tight. These both require substantial memory, and are not required to just run X. Think of these as usability enhancements. 32M probably may not be enough. 64M may be decent, depending on what other applications are being used, and other variables. 128M should be adequate in most situations. 256M or more to be comfortable. File Managers like Nautilus and gmc can also be memory hungry.
Use a lightweight window manger. WindowMaker, BlackBox, IceWM, fvwm (and variants), XFce, all have reputations of performing well with low memory. There are surely others as well. Experiment. fvwm is generally considered the lightest of the light.
A very nice desktop is still very possible even without KDE or GNOME. In fact, most KDE and GNOME applications can still be used even if KDE and GNOME are not running themselves (assuming the right libs are installed).
Don't use fancy themes or backgrounds. Plain and simple is easier on resources. Use a solid color background. Avoid pixmaps or gradients for any kind of background, including menus, title bars, etc.
Use a lesser screen size and color depth. 800x600x16 will not push X as hard and be easier on system resources than higher values. While a ColorDepth of 24 is preferred, you probably will not notice the difference of 16 with the majority of applications.
Some applications require much more memory than others. Some notable hogs are Netscape, Mozilla, office suites, and the Gimp. Netscape is faster than Mozilla (but not as nice). Netscape-Navigator uses less memory than Netscape-Communicator. Close any of these apps when not in use. Use text browsers like lynx or w3m wherever you can, like reading locally installed HTML documentation. Much faster, and much less memory is required.
Also, use text based clients for mail (mutt or pine) and news (slrn or trn). Again, faster and much less memory is used, and these are after all text based protocols at heart anyway.
rxvt uses less memory than xterm, konsole or gnome-terminal.
If you run an X session for long periods of time (like days or weeks), restart X occasionally to free memory tied up as cache.
Disable “backing store” and “save-unders” to
reduce memory usage (performance penalty though). Check
your Window Manager's settings too. See what modules are being loaded
in the “Modules” section of XF86Config as
well. Your installation may have many unnecessary ones enabled, or ones you
can't take advantage of (e.g. “v4l”, aka
“Video4Linux”).
Font servers may provide a slight performance boost by off-loading font rendering to the font server, while freeing the X server to do other things. But, the font server will use a small additional amount of memory as well. So, you can try it either way to see if it makes a difference.
Lastly, RAM is cheap now. Buy some ;-) A new drive too.
RAM is still just too low for X? Check out tinyX: http://www.superant.com/smalllinux/tinyX01.html. Reportedly runs in as little as 4 Meg of RAM.
X is not particularly network friendly. In other words, it is a bandwidth hog. This should not be a problem in LAN situations, but may be if trying to use X over the Internet.
lbxproxy, the low bandwidth X proxy, utilizes various optimizations to improve performance in low bandwidth, or high latency situations. See the man page.
VNC (Virtual Network Computing), has some of the same advantages as X for displaying applications on remote systems, but is more network friendly. Your Linux installation should have both VNC client and server packages available.
Other tips to eek out better performance:
Use xset to speed up the keyboard. This can make the system feel more responsive even if it isn't really. The default always seemed sluggish to me.
renice X to give it a higher priority. Other platforms give the GUI high scheduling priority to achieve better responsiveness. But this is at a cost to other processes. Linux is a blank slate. You might include the font server (if being used), and key KDE and GNOME processes as well.
nice -n -10 X :0
This will not do much on systems that are mostly idle.
This does not work so well with startx since X runs as root, and you are not root, right? So you would have to use something like sudo to have this done automatically.