Consumer-grade desktop for toy-"operational" model running 24/7?

[stef]

I am running a toy "operational" model and consider buying a dedicated consumer-grade desktop for it. The plan is to pick some number of cores (arbitrary, say 6 or 12 cores) and adjust the model configuration and data processing pipeline such that the desktop runs continuously under full load, doing 6-day assimilation-forecast cycles. I will be doing this for a couple of years, refining the model and data processing step by step, all with very low resolution.

The question is: does consumer hardware break if it runs under full load 24/7? Does anyone have experience with running consumer-grade hardware under full load continuously?

I am asking here on the forum because in other forums is always "use a VPS or rent a dedicated server". But I look at e.g. OVH and a 6-core server is $60 per month, which in 4 years is $2880. In VPS the compute is even much more expensive.

My electricity is currently about 250 Euro per year for a 12-core with TDP of about 100 Watts, so for 4 years it's $1000 (and that assumes the TDP is actually drawn continuously, which I doubt.). So I break even with the OVH example above if I spend $1800 on a computer, and the point is I can get a much more powerful machine for this amount than OVH (or any other cloud or dedicated server provider) could offer. A 16-core AMD Ryzen 9 5950X 16C/32T, 3.40-4.90GHz is currently about $500,-.

The difference is, of course, that the server provider provides industrial-grade servers, which are built from the ground up to run 24/7.

Do you have an opinion on this?

Edit: I should say that my internet is already paid for.

[jbettencourt]

I have run a consumer grade desktop (Linux on AMD Ryzen) for one month and a half continuously with a wave model (SWAN) and hadn't any problems or noticed any downgrade with performance, but admittedly, one month is not four years. However, I think that over four years, other considerations such as the stability of the electricity supply or its cost come in to play.

[stef]

Thanks for sharing your experience! Yes, SWAN takes up lots of resources. I guess I'll try to run everything on my existing workstation for now.

[ezaron]

Hi Stef,
The other significant cost to consider with a cloud provider or hardware rental is the data egress. I tried to price this out for a project last year, and the cost of moving the data around for analysis was quite significant. To estimate the price depends on having a rather detailed model for how many bytes per month you expect to move and/or archive, and a good idea of the acceptable latencies.
Also, whether you run the forecast system on your own or you rent hardware in the cloud, it will be important to have a plan for checkpointing and restarts due to system failure.
-Ed

[stef]

Good point, and even with a flat internet rate, I guess the compute nodes should be close to wherever the large data archive is. I guess the largest data set is the output data, assuming one wants to keep it all.

There is a blog post saying it's ok to run 24/7, except the fans deteriorate easily:

https://www.redhat.com/sysadmin/run-my-computer-24-7

[shchepet]

Generally speaking, with limited money you can get better performance vs. $$$ cost, if you with "enthusiast" class PC. Get a high-level gaming mother board. Quad-channel memory system in a MUST. Do not settle with dual-channel memory. Pick reasonably priced Core i7-E or i9 processor (may be AMD Ryzen, but I do not have personal experience with them). Multiple PCI-E slots a kind of excessive, since you are not going to put multiple videocards, however they may be useful to build a decent size and performance RAID system Some time I start writtting a constitution on this subject


http://people.atmos.ucla.edu/alex/compl ... x_raid.htm

[stef]

Get a high-level gaming mother board. Quad-channel memory system in a MUST. Do not settle with dual-channel memory.

Ok! In AMD land that would imply a Ryzen Threadripper I think. They have 4 or 8 channels. In terms of price difference to a server:

Doing a quick search, Lenovo offers a ThinkStation P620 with AMD Ryzen Threadripper PRO 5955WX Prozessor (16C/32T, 4,00-4,50 GHz) for about 4000 Euros (64GB RAM included, no mass storage, no graphics). The CPU has octa-channel memory and a TDP of 280W (yikes). Not really a gaming desktop, but a high-end workstation.

For comparison, I got a quote from a server vendor for a PowerEdge R7515 Rack Server with AMD EPYC 7313P 3.0GHz, 16C/32T for 4500 Euros (32GB RAM, 4TB storage, IPMI and the usual server stuff except RAID). This has 8 memory channels. I guess it has more fan noise than a workstation, in terms of cooling the consultant said I don't need air condition at this level. No idea about the total power draw.

So yes, the workstation seems like a better deal due to the higher clock rate, right?

Intel core i9-X series has been discontinued since 2019:

https://ark.intel.com/content/www/us/en ... ssors.html

Not sure if they have desktop quad- or octachannel CPUs at the moment.

Regarding RAID: Thanks for the link to the blog post! I have not worried about RAID. At the moment I don't need high availability for my toy model. Redundancy via backup (rsync) to a separate disk once per day, perhaps with snapshotting on btrfs, and manually switching drives in case of failure will do fine, I think.

Edit: I browsed through the article, it says RAID it's not just about high availability:

(ii) performance - ability to achieve faster reading and writing speeds beyond of what is achievable by individual disks;
(iii) convenience in handling large volumes of data - from the user point of view array looks just a large single data space.

So I may have to look into it someday.

[shchepet]

AMD Ryzen Threadripper-pro-5995wx may be a very attractive and promising option

https://www.amd.com/en/products/cpu/amd ... pro-5995wx

but I have no personal experience with them to advise.

In any case, memory bandwidth is always a bottleneck, so the more memory channels you have the better. Attached is timing tests for running ROMS Soliton problem on 1156x384 grid on a fairly modern desktop machine with Intel Core i5 10500 CPU, dual-channel DDR4 memory. Horizontal axes is partition (number of subdomains with various partition policies); vertical axis is computing speed.

The boundary between PC and Workstation is rather vague nowadays. What Dell and HP call workstation for advertising purposes is basically a server type CPU (support of ECC memory) with decent videocard and and sound added (server motherboards have low-end video on-board and do not have sound cards). Thus, for computing purposes workstation are very similar to servers.

Enthusiast PC have high-end desktop (hence no ECC memory support) CPUs, do not have dual-CPU capability, but usually have faster MHz. Roughly speaking, look for no_of_cores*MHz against the price, and chose best on what you can get.

[stef]

Good to know, thanks for all the great advice!

[shchepet]

...and, of course, all the disappointing experience from the past still applies as it was just yesterday night viewtopic.php?p=7771

[stef]

Wow, the information on the thread you link to is all new to me! I looked at your figure in this thread and wondered why there are so many tiles. Now I see it's explained in the thread. My hopes are up that I can get a significant speedup by configuring everything properly. Hernan is right in the post where he says this should go into a wiki page. I had no clue about this.

[xbrogo]

Hi

In the CFD community the AMD-EPYC processors seems to be favored. They are a bit pricy but good performance for CFD (i.e. good on memory bandwidth).

Göran

[stef]

Ok, thanks for the tip! Here is a tomshardware article about the AMD threadripper strategy [1], if I understand correctly the prices for the workstation CPUs with quad- and octacore memory do not differ much from server CPUs nowadays. There used to be an "enthusiast" line which they don't offer anymore.

https://www.tomshardware.com/reviews/am ... cpu-review

Anyways, after reading the thread shchepet linked to, I won't buy anything soon and instead try to understand my current workstation.

[jbettencourt]

There's great info in this thread (and in the links provided). Has anybody had experience with ROMS on Apple Silicon? I use a M1 MacBook Pro to setup ROMS configurations and run them on Intel chips, but I'd really like to know how do the Apple chips compare to Intel when it comes to production runs.

Best,
Joao

[arango]

We updated the code last week to use the NAGFOR compiler directly with Apple Silicon. See Darwin-nag.mk. David Robertson is benchmarking ROMS on the new Macs and comparing it with ifort.

[jbettencourt]

Good to know that there's NAG Fortran compiler for Apple Silicon Macs. I've been using an experimental version of gfortran 11 (https://github.com/fxcoudert/gfortran-f ... arm-alpha1). But since I don't have access to ifort, I can't compare both.

[jpringle]

Just a quick callout to a past post of mine viewtopic.php?t=4665 on performance versus memory bandwidth.

I still find this to be true with the latest AMD hardware for ROMS -- The 16 core AMD 5950x is a beast for everyday work, but with its two channels of memory I can't get it to perform faster by using more than 6 cores, while my ancient first generation thread ripper parallelizes much more efficiently (while being slower per core), and there is a real performance/$ sweet spot with epyc for ROMS at 2 core/memory channel -- again, see my post above.

If you have a place where you can put a noisey server, you can get very good and relatively cheap performance for a dual 16 core epyc server style machine. The biggest issue is the noise they make... I am sure someone has tried to put in in a desktop form factor, but am not sure. Reach out if you want details.

jamie

[arango]

Throughout the years, there has been excellent discussions and different point of view about benchmarking ROMS on various hardware. Perhaps, it is time to summarize that information in wikiROMS for easy and centralized access in the future. Thus, I am adding this task to our TODO list.

[rduran]

If you have a place where you can put a noisey server, you can get very good and relatively cheap performance for a dual 16 core epyc server style machine. The biggest issue is the noise they make... I am sure someone has tried to put in in a desktop form factor, but am not sure. Reach out if you want details.

Regarding the noise, I am wondering (and hoping) that liquid cooling could solve that problem. Any experience regarding this from anyone? And if it is possible to keep it silent enough with liquid cooling for the processor, would add GPU(s) imply too noisy again?

Rodrigo

[stef]

Just a quick callout to a past post of mine viewtopic.php?t=4665 on performance versus memory bandwidth.

Thanks, I haven't seen this!

ROMS scales well with increasing number of MPI processes (eg. NtileI*NtileJ) until the number of processes exceeds twice or three times the number of memory channels -- two to three processes per memory channel is about as good as it gets.

That's very interesting, I have a 12 core Ryzen 9 3900X, and have been working with 6 cores for the past years, like you mentioned in your above post. However, I thought it was because my domain is so small, and that with a larger domain, I would be using the additional cores. Never really measured it. So regarding my plans for the consumer-grade machine (running ROMS exclusively except some pre- and postprocessing), the conclusion would be to by a 6-core Ryzen, as a higher core number would be a waste of cores with 2 memory channels. A Ryzen 5 5600G, 6C/12T, 3.90-4.40GHz would cost me roughly 180 Euros, so that would be a great saving relative to the 16-core machine I mentioned earlier. This thread helped me a lot - thanks again!