Become a TechRadar Insider
TL;DR
- 5th Gen AMD EPYC™ processors are optimized for AI CPU inference
- AMD EPYC CPUs excel for small and medium AI inference models, as well as LLMs
- AMD has technical collaborations with major AI companies like Hugging Face, AI Alliance, and OpenAI.
- Servers powered by AMD EPYC CPUs are available from multiple well-known vendors like Dell, HPE, Lenovo, and Supermicro
- AMD EPYC CPUs use x86 architecture, making them compatible with most enterprise software
When it comes to thinking about AI transformation, there’s a temptation for businesses to focus on the software layer – chatbots, agents, document creation, and so on – above all else. For businesses that are really serious about their AI journey, hardware is equally important.
Servers, whether located in an on-premises data center or a colocation provider, are increasingly the backbone of AI efforts as companies realise they can’t rely solely on public cloud AI offerings. Organizations that want to run their own language models on-premises need the appropriate infrastructure to do so and that means getting the right server CPU for the job.
AMD EPYC CPUs are optimized for enterprise AI CPU inference and excel for small and medium language models of the kind a business might run on-premises. They’re also available as part of numerous server options from well-known enterprise hardware vendors.
Read on to find out how AMD EPYC chips can help in businesses’ AI transformation journeys.
What is AMD EPYC?
AMD EPYC is the AMD line of server and data center processors. 5th Gen AMD EPYC™ server CPUs, also known as AMD EPYC™ 9005 series server CPUs, are the most recent generation of the EPYC™ family.
Are 5th Gen AMD EPYC processors performant enough for AI?
Yes, AMD EPYC™ 9005 server CPUs are not only performant enough for AI, they’re optimized for enterprise AI CPU inference and can also host GPU-accelerated systems efficiently.
5th Gen AMD EPYC processors offer outstanding performance and efficiency, with high CPU frequencies, high core counts, large cache, and high memory bandwidth.
Compared to alternatives like Intel Xeon chips, AMD EPYC 9005 series chips stand out from the competition. For example, two-socket servers using 192-core AMD EPYC 9965 CPUs can achieve up to 1.9x the inference throughput when running XGBoost with the Higgs boson data set compared to two-socket servers using 128-core Intel Xeon 6980 CPUs.
Additionally, according to internal AMD testing across a range of models, a 2P high-frequency AMD EPYC 9575 CPU-based server with eight GPUs can deliver up to 6% higher overall inference throughput compared to a similar eight GPU server powered by two Intel Xeon 6960P CPUs, as well as 13% faster time-to-first-token.
Are AMD EPYC processors only useful for companies wanting to run large inference models and GPU-accelerated workloads?
No. In fact, AMD EPYC processors excel for small and medium AI inference models.
Small inference models, such as Hugging Face’s SmolLM2-1.7B, Mistral Ministral 3 3B, or Meta’s Llama 3.2-1B, typically have between one and four billion parameters, according to a Hugging Face blog post.
Medium AI inference can extend into tens-of-billions of parameters: Qwen’s QwQ-32B, has 32 billion parameters, for example.
While small inference models and medium inference models are much smaller than large language models (LLMs) – which can measure hundreds-of-billions or even, in the case of GPT-4, in excess of a trillion parameters – they are still hefty. Any hardware running them needs to be capable of doing so properly, which is where AMD EPYC 9005 server CPUs shine.
Are AMD EPYC processors accredited to work with any major LLM suppliers?
AMD has technical collaborations with well known AI companies, including:
- AI Alliance
- Hugging Face
- Lamini
- OpenAI
AMD also participates in cross-platform open-source initiatives like PyTorch, TensorFlow, Triton, OpenMP, Open XLA, and MLIR.
Which OEMs offer servers with AMD EPYC CPUs?
When deciding how to implement AI in a business, it’s important to consider which hardware vendor you want to use. Perhaps you have an existing provider you want to stay with, or maybe you’re willing to shop around.
Whatever the scenario, if you want to take advantage of the AMD EPYC CPU’s features, the good news is many enterprise OEMs sell servers featuring AMD EPYC CPUs, including:
Are AMD EPYC CPUs easily compatible with other hardware and software environments?
Yes, AMD EPYC CPUs are compatible with most hardware and software that you’ll find in the data center already.
AMD EPYC server CPUs use x86 architecture, which is found in data center environments as well as PCs and some laptops. This means that when moving from a rival CPU such as Intel Xeon to AMD EPYC server CPUs, there’s no need to retool workloads to run on unfamiliar architecture. This is important for standard software, of course, but for rapidly evolving AI systems a smooth migration to an AMD CPU-powered platform from one that isn’t is vital.
Once AMD CPU-powered servers have been acquired, tested, and are ready to go into production, the migration process is greatly simplified for IT admins by the compatible x86 architecture. Start by moving workloads that won’t cause disruption if there are problems, then gradually move virtual machines and production apps into the new environment. Finally, after completing at least one trial run in a sandbox, move workloads like ERP systems, database applications, or anything else considered mission critical over to the AMD EPYC CPU-powered servers.
What additional features does AMD EPYC have?
In addition to everything already mentioned, AMD EPYC 5th Gen CPUs are also optimized for confidential computing and introduce Trusted IO to the Infinity Guard feature set, extending the trust boundary to external devices such as storage, SmartNICs and accelerators*.
All AMD EPYC 9005 server CPUs are officially certified for NIST CMVP FIPS 140-3 Level 1, which is crucial for government agencies and businesses requiring validated and tested security measures for sensitive information.
* AMD Infinity Guard features vary by EPYC™ processor generations and /or series. Infinity Guard security features must be enabled by server OEMs and/or Cloud Service Providers to operate. Check with your OEM or provider to confirm support of these features. Learn more about Infinity Guard at https://www.amd.com/en/technologies/infinity-guard
If you think EPYC is the right compute solution for your business, find out more on the AMD website: US readers click here and CA readers here.