For the last 6 years, my ocean modeling needs have been well met by a 6 node cluster; each node a dual Opteron-250 with a single core for a total of 12 cores. This system has enough power to meet my modest process-oriented modeling needs; typical runs for me have a grid size of 380 by 90 by 25 and run for 200,000 timesteps. For the last 5 years, it was rock solid. However, it is starting to get old, and I am spending more time replacing power supplies and the like, and less time doing model runs.
I am thinking about how to replace it with a system of comparable power. I am attracted to a single box with dual processors and many cores so that I don't have to deal with the complexity and cost of a cluster (interconnects, etc). I also think that being able to use open-mp will allow us to tile the problem in a way that is more cache-friendly than using MPI on a cluster with ROMS 3.
In the past, I have been guided by Sasha Shchepetikin poorman's super-computer documents, and more recently his posts in viewtopic.php?f=17&t=2001&p=7771 . I am considering two systems: dual 6-core 2.6Ghz Xeons (X5650) or dual 12-core 2.2Ghz 12 core Opterons (6174).
- Is there a different sort of system I should be considering?
- Does anyone have experience with these systems? Do dual core systems of this sort have enough memory bandwidth to keep the cores fed?
- I am particularly concerned that the Opterons may have insufficient memory bandwidth to keep the cores supplied with data. Can this be alleviated by choosing appropriate tile sizes and using open-mp?
- Is the Intel compiler still the compiler of choice for these systems? For both the Opterons and Xeons?
- For my size problem, is the "big-box" single computer preferable to a small cluster?
- Are there any other gotchas to consider in specing such a system (For example, Sasha has stressed the importance of making sure the memory modules match the memory channels for each system).
- Are there some motherboards to be sought/avoided?
- Is the choice of memory (and its timings, etc) important?
- What is the best linux kernel/distributions for running ROMS? (I don't care about deskstop stuff; I am not sure I would even run an Xserver on it.) I am most concerned about getting a kernel optimized for throughput and not interactivity.
and thanks,
Jamie Pringle
University of New Hampshire