Constructing an HPC Cluster for weather modeling requires expertise and loads of preparation. It’s important to know both the scale and scope of the workloads to be performed before beginning to build out any HPC Cluster platform. It’s also important to note which specific weather modeling applications will be utilized. With WRF being one of the most popular and widely used applications in the weather modeling industry, we’ve been able to develop years of experience and expertise building WRF specific HPC Clusters.

WRF is extremely powerful and requires significant computing resources to operate efficiently. As a result, organizations looking to utilize WRF see the most success when deployed in an on premise environment.  This is especially true when the volume of the runs, required resolutions and sizes of the models are significant. Running the same WRF models using cloud resources instead of on premise HPC Clusters would cost anywhere from 300% to 500% more — without the guarantee of answers in a timely manner.

With so many things to consider and weigh when building an HPC Cluster for WRF, we outlined the five most important factors to keep front of mind to help.

5 Things to Consider when Building an HPC Cluster for WRF

1. Determine the Number of Processor Cores Required. This step is usually easier for those who have experience running WRF on an existing HPC cluster, but if that’s not you, no need to worry. Instead, try to determine the complexity of the models, resolution required and number of runs per day/week/month/year. By doing so, you should be able to determine with some degree of certainty the size of the HPC Cluster.It’s also important to select the right processor manufacturer and model during this step. A good rule of thumb to follow is to select a processor that offers higher clock speeds, (i.e. above 2.4GHz), and worry less about the highest number of cores per processor. This will help ensure you are not degrading performance by exceeding the system’s memory bandwidth capabilities. At PSSC Labs, we have primarily used Intel^® Xeon^® Scalable processors to date, but we’re starting to see more and more interest in AMD EPYC ^TM processors, due to lower cost and the potential for higher clock speeds. Our initial performance results when comparing the two different types of processors do not provide a clear cut winner. One more important note is that using the Intel^® Cluster Studio XE Compiler Suite can provide a huge performance improvement; tipping the scales in the Intel^® Xeon^® processor favor if you are on the fence choosing between the two processors.
2. Select the Amount of Memory per Core. At the absolute bare minimum, we recommend 2 GB memory per core, though we don’t deliver many HPC Cluster systems with less than 4GB memory per core. Going higher, say to 8 GB, would likely be overkill. Keep in mind that this step of the process is really about configuring the memory for maximum memory bandwidth, as memory access will have a huge impact on the overall cluster performance. With WRF, it’s all about moving data in and out of the processors as quickly as possible, and memory bandwidth has a tremendous impact on performance.
3. Build the Fastest Possible Network Backplane Your Budget Allows. This step should be front of mind for larger clusters (i.e. several thousand cores). As stated in the above step, getting the data to the processor as fast as possible is critical and having the highest speed network backplane for an HPC Cluster significantly helps in this effort. We typically employ Intel^® Omni-Path and Mellanox^® InfiniBand^® when building HPC Clusters for WRF. With NVIDIA’s Mellanox’s latest 200 Gb/sec HDR Infiniband^® backplane, Mellanox has taken the lead over Intel^® Omnipath^® which tops out at 100 Gb/sec. Mellanox does offer a cost effective 40 port 200 Gb/sec HDR Top of Rack switch, which is becoming more and more of a standard for our HPC Clusters.
4. Consider Using a Parallel File System to Increase Performance and Offer Scalability. We’ve worked with several parallel file systems, including HDFS, GlusterFS and LusterFS. Each of these has their own pluses and minuses but they all offer a significant improvement over a standard NAS or NFS storage server. With a parallel file system, you’re essentially spreading the load of data access across multiple storage nodes and targets. Our Parallux Storage Clusters have achieved over 50 GB/sec sustained Read / Writes. This represents a huge improvement over stand-alone NAS servers that max out around 2.5 GB/sec. Faster access to data means reduced computing times because you’re better able to keep the processors maxed out with data access.
5. Build with the Future in Mind. Like most HPC applications, WRF can consume all the computing resources you can throw at it and still keep asking for more. We always allow room in our HPC Clusters to double or triple the number of processor cores over time. Adding nodes to an existing HPC Cluster is not complicated. Using simple tools, like Clonezilla, will allow you to keep expanding your cluster as needs grow and your budget allows.

Building an HPC cluster for WRF doesn’t have to be overwhelming, especially when working with the an experienced and knowledgeable HPC Cluster manufacturer. PSSC Labs has 25+ years of experience working closely with clients to determine their needs and architect a custom HPC Cluster. For more information please visit https://pssclabs.losangeles.dev.buckupstudio.com/solutions/weather-modeling.

For questions regarding our HPC Clusters for WRF and other weathers models, please feel free to contact us at 4sales@site-a.com or (949) 380-7288.

Graphics Processing Units (GPUs) are built to speed up the process by which a computer creates and renders images intended for output to a display device. In addition to this, GPUs also ensure that graphics are displayed properly and work to enhance graphic performance. GPUs live in all kinds of technology hardware, from computers to laptops to cellphones and gaming consoles.

When it comes to working with a vendor to build a custom high performance system, selecting each component specifically for your needs is extremely important. We work with our clients to determine their needs and help them pick each component specifically for their workloads, but it’s always great when our clients have some familiarity with the pieces of their system. Below we breakdown some important things that we look at when helping our clients pick the right GPU.

NVIDIA has recently released a new series of graphics cards, the GeForce RTX™ 30 Series. This series is the second generation of the NVIDIA RTX PC gaming platform, and regarded as the fastest discrete graphics memory graphics card on the market today. Built with enhanced Ray Tracing Cores and Tensor Cores, new streaming multiprocessors, and high-speed G6 memory, this structure gives users the power needed to rip through the most demanding games.

GPU Options to Consider

1. GeForce RTX™ 3070

   Base/Boost Core Clock: 1400/1700 Mhz
   Bandwidth (GB/s): 1000
   Memory Size (GB): 8
   NVIDIA CUDA Cores: 5888

2. GeForce RTX™ 3080

   Base/Boost Core Clock: 1440/1710 Mhz
   Bandwidth (GB/s): 1000
   Memory Size (GB): 10
   NVIDIA CUDA Cores: 8704

3. GeForce RTX™ 3090

   Base/Boost Core Clock: 1500/1730 Mhz
   Bandwidth (GB/s): 1000
   Memory Size (GB): 24
   NVIDIA CUDA Cores: 10496

4. NVIDIA A100

   NVIDIA GPU for HPC built the A100 GPU specifically for scientific computing, graphics, and data analytics in data centers. Even having launched this product amid a global pandemic, NVIDIA CEO Jensen Huang stated, “It’s our best data center GPU ever made, and it capitalizes on nearly a decade of our data center experience.”
   Researchers and engineers alike need to be able to analyze, visualize, and turn massive datasets into actionable insights. The problem is that these datasets often are scattered across multiple servers and therefore, the entire process gets bogged down. With the A100, the right amount of compute power, memory, and scalability is delivered to help organizations tackle their massive workloads.
   The NVIDIA A100 has more than 54 billion transistors. It’s the world’s largest 7nm processor. A100 can also efficiently scale to thousands of GPUs or, with NVIDIA Multi-Instance GPU (MIG) technology, be partitioned into seven GPU instances to accelerate workloads of all sizes.

   Base/Boost Core Clock: 1430/1480 MHz
   Bandwidth (GB/s): 1555
   Memory Size (GB): 16 or 32
   

5. NVIDIA V100

   The NVIDIA V100 GPU is powered by NVIDIA Volta architecture and comes in both 16 and 32 GB configurations. This GPU also offers up to 100 CPUs, meaning data scientists, researchers, and engineers can now spend much less timing working to optimize memory usage and more time on their important artificial intelligence efforts. Even better, with 640 Tensor Cores, Tesla V100 is the world’s first GPU to break the 100 teraFLOPS (TFLOPS) barrier of deep learning performance.
   Base/Boost Core Clock: 1246/1380 Mhz
   Bandwidth (GB/s): 897
   Memory Size (GB): 16

6. GeForce 2080 Ti
   

   GeForce GPUs are another family of GPUs developed by NVIDIA. These GPUs are popular in gaming systems due to their ultra-fast processing speeds. In fact, this GPU in particular is considered the world’s ultimate gaming GPU. It’s release into the market marked the introduction of NVIDIA’s Turing microarchitecture, the first in the industry to implement real-time hardware ray tracing in a consumer product. With a base core clock speed of 1380 MHz, the GeForce 2080 Ti is quite powerful and delivers a solid framerate, even when advanced features are enabled.

   Base/Boost Core Clock: 1350/1545 Mhz
   Memory (MT/s): 14000
   Bandwidth (GB/s): 616

7. Tesla T4

   The Tesla T4 GPU was launched in September of 2018. NVIDIA built this GPU in an effort to accelerate workloads specifically in the HPC, deep learning, machine learning, data analytics, and graphics spaces. Tesla products are primarily used in simulations and large scale calculations, as well as for high end image generation in professional and scientific fields.
   The Tesla T4 is a rather large chip but a single-slot card, so it doesn’t require an additional power connector, as its power draw is rated at 70W maximum. Unlike the fully unlocked GeForce RTX 2080 SUPER, which uses the same GPU but has all 3072 shaders enabled, NVIDIA has disabled some shading units on the Tesla T4 to reach the product’s target shader count. It features 2560 shading units, 160 texture mapping units, and 64 ROPs. Also included are 320 tensor cores which help improve the speed of machine learning applications.

   Base/Boost Core Clock: unknown/1455 Mhz
   Bandwidth (GB/s): 900
   Memory Size (GB): 16 or 32
   

8. Quadro RTX 6000

   Quadro has long been the de facto standard for enterprise desktop graphics for digital designers and artists, but the Quadro RTX 6000 also stands out as the world’s first ray tracing GPU. With the launch of the Quadro RTX 6000, significant graphic advancements have been brought to professional workflows. In general, though dependent on application, this GPU is faster than the GeForce 2080 Ti, making it arguably the fastest graphics card in any environment.

   Base/Boost Core Clock: 1440/1770 Mhz
   Memory (MT/s): 14000
   Bandwidth (GB/s): 672
   Memory Size (GB): 24

9. Quadro RTX 8000

   Optimized for workstation applications like CAD and 3D modeling, artists and designers are able to push the boundaries of possibilities in their line of work with the Quadro RTX 8000 graphics card. This Quadro RTX product is built to work with the largest and most complex ray tracing, deep learning, and visual computing workloads.

   Base/Boost Core Clock: 1440/1770 Mhz
   Memory (MT/s): 14000
   Bandwidth (GB/s): 672
   Memory Size (GB): 48

   As a custom manufacturer of high performance computing and big data systems that use GPUs and graphics cards like those listed above, we know how important it is to build your system with the right components. That’s why we work closely with our clients to understand their workflow and data needs, then build up the system that will work perfectly for their organization.

   If you have questions about these components or any of our other products, please feel free to contact us at 4sales@site-a.com

Building a high performance computing cluster for your specific needs used to be easier said than done, but we’re making strides to change that. With organizations needing their own HPC and big data systems now more than ever before, it’s important that the process of identifying what kind of system you need be as simple as possible, especially those in the engineering space that are working day after day to solve the world’s problems. This means that engineering organizations, particularly Ansys users, need systems that are designed to run exactly as they need them to, for their unique purposes. Here are five things Ansys users should consider when configuring a custom HPC cluster:

Analyzing User Needs

How many users? What type of jobs will they be running? Are some users priority over others? Being able to set user expectations on utilization of the system cannot be overstated. If you have hundreds of users, then be sure to budget and build a machine that can easily meet this demand. If some users have higher priority than others, make sure this information is available to the manufacturer, as the job scheduler will need to be setup and configured correctly.

Technical Capability of Users

Are all users comfortable with a consolidated HPC environment? Are they mostly Windows or Linux? Will there be a dedicated systems administrator? The answers to these questions will have a direct impact on the direction you take when it comes to choosing the right HPC cluster vendor.

HPC Cluster Hardware

Most Ansys users typically require a fair amount of computing power. This means that selecting the right number of cores, memory and high speed network backplane is critical to a successful deployment. We’ve found that fewer cores per node paired with a higher clock speed is typical because of the way that Ansys licenses their software applications. AMD EPYC CPUs offer a significant jump in GHz speed per core, while still maintaining reasonable cost.

With the minimum amount of memory being 2 GB memory per processor core, and Ansys users tending to use 6 GB memory per core or more, it’s important that memory configuration be optimized for maximum bandwidth.

Additionally, one node should be dedicated to visualization. Be sure to include a NVIDIA Quadro GPU in the “Head Node” or dedicated “Viz Node” to meet this need.

Determining Budget

In order to maximize your budget, this needs to be determined as early as possible. Establishing budget early on means that you are able to prioritize which components of your system are most important. If you need a machine with thousands of cores, but are operating under a limited budget, your vendor can help you identify trade-offs within the system that can make this possible.

Selecting Vendor

There are many options when it comes to selecting an HPC vendor, from cloud providers to Tier 1s to smaller, specialized HPC manufacturers. For users that just need to run a jump or two once in a while, cloud resources may be enough. But often times, these same users reach a certain usage level that makes the cost of cloud computing unreasonably high, and that usage level threshold is often much lower than users anticipate.

Nearly all cost calculators will eventually point you in the direction of purchasing a dedicated piece of hardware if you accurately account for the workloads you’ll be running on your cluster. In fact, most buyers find that the cost of buying a dedicated HPC cluster pays for itself when compared to 6-8 months of consistent cloud-usage. This means that most Ansys users end up turning away from cloud computing options during this discovery phase, and rightfully so. They then begin comparing the trade-offs of working with a Tier 1 manufacturer versus a company that specializes in delivering HPC clusters to Ansys end users.

With years of experience providing high performance computing clusters to Ansys end users, our engineers here at PSSC Labs listen to the specific needs of our clients and then work with them to customize a solution, within any time and budgetary constraints. Our PowerWulf ZXR1+ HPC Cluster is the cluster that many of our Ansys users end up purchasing, and for good reason – it’s application-optimized, scalable, and delivered production ready.

All in all, there are several things to consider when selecting or building a custom HPC cluster. That’s why it’s even more important to work with the right partner – one that can listen to your unique business needs and help you build a system that will perform exactly as you need it to, without the sky high costs of Tier 1 manufacturers.

To request a quote for the PowerWulf ZXR1+ HPC Cluster, or any of our other systems, click here. 

High performance computing (HPC) generally refers to the practice of aggregating computing power in a way that delivers higher performance than one could get from a typical workstation or desktop computer, in an effort to solve problems in a variety of fields – business, science, engineering, and more. Over the past few decades, high performance computing has changed the way we perform research by speeding up the processes by which we can collect, manage, store, process, and analyze information. It’s fair to say that without advancements made by HPC products, research in many areas would have come to a screeching halt many years ago, and this is particularly true of the life sciences industry.

The life sciences space is dominated by data analytics. Almost all HPC work is comprised of large data sets that have extensive storage and networking requirements to handle real-time influxes from various sources. This means systems working on life science research projects need a surplus of processing power.

How HPC is Changing the Life Science Industry

The availability of high performance computing systems in the life science industry presents an opportunity of economic competitiveness and scientific discovery. With HPC being the technology at the forefront of scientific research, it’s truly positioned to help life science organizations pioneer new solutions to the world’s heath problems – something we’ll all find so relevant in these times.

HPC and big data technologies allow research organizations the opportunity to drastically accelerate discovery life cycles across the life science industry, through scenarios such as:

• Chemistry: bio-chemistry, molecular modeling, etc.
• Bio-engineering: agricultural engineering
• Genomics: complex disease genomics, agrigenomics, oncology, microbial genomics, etc.
• Biology: molecular biology
• Pharmacology: pharmacokinetics and pharmacodynamic modeling, drug discovery, etc.
• …and so many other disciplines within the life science space

As artificial intelligence continues to be adopted by organizations of all industries, it becomes more and more clear how significant this kind of technology is, particularly when applied to the life sciences research field and especially in the widespread pandemic like that caused by COVID-19. AI along with medical imaging is being harnessed in the fight against COVID-19 in diagnosis, treatment, and monitoring. With the help of artificial intelligence, researchers are also developing new tools that can help with early detection of the virus, plus personalized therapies for those infected.

HPC for Life Science: Cloud vs. On-Premise

While it’s safe to say that nearly all life science research organizations have adopted high performance computing as a means of managing their data, there’s still the argument of where to manage and store that data – through either on-premise or cloud computing resources. Some companies look into moving to the cloud in an effort to save on storage and processing costs, but what they’re not aware of is that the cost of storing data on the cloud gets more expensive as your data sets grow. Seeing how life sciences research organizations are forever learning and ingesting new data sets, it only makes sense that these scientific leaders are seeing their storage needs grow as well. With an on-premise data environment, data is stored at a fixed cost – the cost of the purchased system. There’s no sky high prices associated with accessing the very data that is already owned, something that cloud services can’t provide.

Other discrepancies between cloud providers and on-premise are those of security and reliability. With data stored in the cloud, everything is shared and no one, except the cloud service provider, owns anything. That means that the hardware, and physical resources inside of it like CPU cores, storage drives, and network interfaces storing your data are likely being shared by other organizations as well, organizations that could be direct competitors or even being investigated by government agencies due to questionable behavior.

In respect to reliability, storing data in the cloud means that organizations are at the mercy of their cloud provider. If that providers systems are down, there’s nothing to do but sit and wait through the restoration process. When data is stored on-premise in a data warehouse, any downtime would be rare, but more importantly, the hardware is physically accessible should any issues arise. Hardware providers also have dedicated support engineers to help address issues and minimize any downtime.

With the many scientific advancements that have come to light over the past few decades of widespread high performance computing in the life science space, it’s no secret that HPC will be the game-changing technology needed to solve our world’s health problems. We’re just here to make sure these organizations fighting on the frontlines have the tools and information they need to be successful, because a healthier world is one we all want to see.