Laatansa Imroni, Author at Efison Lisan Teknologi https://efisonlt.com/author/laatansa/ Computation for Everybody Thu, 28 May 2026 10:38:46 +0000 en-US hourly 1 https://wordpress.org/?v=7.0 https://efisonlt.com/wp-content/uploads/2020/03/cropped-efison_logo_orange-skuer-32x32.png Laatansa Imroni, Author at Efison Lisan Teknologi https://efisonlt.com/author/laatansa/ 32 32 Our Experience with MSI EdgeXpert – Nvidia DGX Spark GB10 https://efisonlt.com/our-experience-with-msi-edgexpert-nvidia-dgx-spark-gb10/?utm_source=rss&utm_medium=rss&utm_campaign=our-experience-with-msi-edgexpert-nvidia-dgx-spark-gb10 Thu, 28 May 2026 07:48:40 +0000 https://efisonlt.com/?p=1974 “Agentic AI”. That’s the key phrase. Everybody is talking about how we should utilise AI on our life. Invest on the skill, buy your own hardware, they said. And this one is probably the best gateway hardware out there. Behold! A mini AI supercomputer (Nvidia’s words, not mine) Such a big statement, I thought. But… Read More »Our Experience with MSI EdgeXpert – Nvidia DGX Spark GB10

The post Our Experience with MSI EdgeXpert – Nvidia DGX Spark GB10 appeared first on Efison Lisan Teknologi.

]]> “Agentic AI”. That’s the key phrase.

Everybody is talking about how we should utilise AI on our life. Invest on the skill, buy your own hardware, they said.

And this one is probably the best gateway hardware out there.

Behold! A mini AI supercomputer (Nvidia’s words, not mine)

Such a big statement, I thought. But after a short exploration, this thing is really a minified supercomputer in many different aspects. It has this enterprise-level of software support, clustering, network interface, and also the emphasis on being an AI server, not just a mere mini PC.

I won’t rewrite the full specification table here. You can see it yourself on MSI EdgeXpert page or Nvidia DGX Spark page.

But let me tell you all the interesting parts.

At a Glance

Ignore the scratches. Not mine to blame. It has almost a half of its height dedicated for an intake grill.
Back view. On top is the exhaust grill. All the physical interfaces are on the backside. Yes, including the power button.
USB ports
4*USB type-C, no USB type-A in sight.
The ConnectX-7 200 Gbps ports for a better clustering support compared to any mini PC I’ve ever seen.

The omission of power LED or any power-on indicator is kinda heartbreaking, tho.

Hardware Overview

This system is using a custom multi-chip package SoC called Nvidia GB10. The GB10 is consisted of (should be) Grace Arm CPU and Blackwell GPU. Nvidia didn’t really mention the specific name of both the CPU and GPU so I’ll put the specification below to represent the number of the compute unit count instead.

ComponentSpecification
CPU10-core Arm Cortex-X925, 10-core Arm Cortex-A725
GPU48-SM Blackwell
Storage1 TB NVMe
Memory128 GB 256-bit LPDDR5x unified
Memory Bandwidth273 GB/s
Ethernet10 GbE RJ-45
High-Speed NetworkConnectX-7 NIC @ 200 Gbps
Wireless ConnectionWifi 7, Bluetooth 5.4
Power Supply240 W AC-to-DC power brick via USB type-C
BMC/IPMINone

As it stands, my main complaint is the lack of baseboard management controller (BMC) to remotely monitor the state of the hardware, connect to a virtual KVM to the server, and to control the power state, exclusive from the OS. This way, you can only get the monitoring of the unit after you boot into the OS. Well, unfortunately Nvidia decided it is not supercomputer enough to bear the right of having a BMC.

Software Overview

This one is a more interesting one. The OS is called DGX OS officially, but it’s actually an Ubuntu 24.04 with some “custom sauce” from Nvidia.

neofetch of DGX Spark.

The “custom sauce” lies on the repository being used for delivering packages optimised for the GB10 which are maintained and deployed by Nvidia.

Nvidia repo for DGX Spark.

It comes with GNOME Desktop Environment if you fancy using it as a normal desktop PC. You can install various software through command line or Software Center in a typical Ubuntu desktop fashion.

Webcam works, too, if you feel like wanting to plug a webcam.

As it is a mini server, it has this cute quirk, which is you’ll find no sleep (or hibernate) option. You can only set the power saving to blank the screen after being left idle for some time. Well you don’t want your server to suddenly sleep in the middle of serving, no?

No sleep. Sleep is for the weak. Excuse my keyboard (Bridge75) for the photobomb.

Being an AI Server: Default Software

After first boot and setup, you’ll get yourself a default dashboard called DGX Dashboard which can be accessed by a browser. The default address is http://localhost:11000 which can only be accessed locally from the machine itself. But you can easily redirect it with a proxy or tunneling just fine.

DGX Dashboard

As can be seen above you can easily see simple statuses of the hardware and launch a JupyterLab which would then can be used as PyTorch or TensorFlow development environment just as easy. From there you can use the underlying hardware both CPU and GPU to develop or test your various Jupyter workflow.

From the same dashboard we can click on the Updates menu to… update. Nvidia recommends updating from this dashboard instead of using CLI, and you will also get your firmware or UEFI update from this same interface.

DGX Dashboard: Updates

Continuing right there is a Settings menu which can be used to change the hostname and enable/disable telemetry.

DGX Dashboard: Settings

The next three buttons on the right are external links in which you can interact with:

Docs: Product Documentation and User Manual

Here you can look for everything you need to know about DGX Spark down from its hardware, software, configuration, release notes, system update guide, how to get support, to legal information. This is as complete as you want it to be. Personally I found that this is the most complete manual ever written for a mini PC. Oh wait it’s a mini supercomputer, not just a mini PC.

Docs button will bring you to DGX Spark User Guide site.

Forums: Discussion Boards

Do you ever feel like missing a bulletin boards? Or interacting with a bunch of geniuses around the world who have the same systems? Afraid not! This forum is as lively as it can, with Nvidia moderators also actively participates in the discussions. Maybe someone has a crazy idea, maybe someone has a solution for a particular problem, you name it. Lovely.

Lively forum.
Maybe a little bit too lively. But what is life without genius madlads around?

Resources: Guides for Deploying AI Softwares

For me this is one of the killer features. This is the guide for building, running, and deploying any popular AI software, some with prebuilt container images or models built by Nvidia themselves. Again, personally this has been the best resources provided by first-party for users to have the least possible headache of running any AI software they want to use.

They encourage you to build any AI software in the easiest way possible.
Example: vLLM serving guide.
They have their ready-to-run models, if you prefer.
You can also use Nvidia’s prebuilt container image from the NGC.

CPU (and Memory) Performance Benchmarks

Here I tested some benchmarks to portray how fast it is in general tasks.

Geekbench 6

It slightly beat my desktop PC which has a bigger cooler and consumes more power.

Geekbench 6 result against an 8P core i7-12700K.

Phoronix Linux Kernel 7.0 Compilation

Unfortunately it fell short in compilation performance. Personally, I also found this machine to be quite slow during Nunchaku wheel compilation which I will explain later on.

Compilation time of Linux Kernel 7.0.

It is slower than an aging 8-core 8-thread i7-9700. Ouch.

Linux kernel 7.0 compilation time comparison on various different CPUs.

Phoronix SVT-AV1 4.0

Similar picture painted when tested against SVT AV1 encoding using CPU.

AV1 encoding speed using SVT library.

It’s only marginally faster than an aging 4-core 8-thread mobile CPU i7-8550U.

SVT-AV1 performance comparison on various different CPUs.

Phoronix 7-zip 26.01 Compression and Decompression

This one yielded a better outcome. My theory would be the large memory bandwidth helps with the compression/decompression performance by much and feeds the cores with the necessary data nicely.

7-zip 26.01 file compression speed.

Only slightly lower than another mini AI machine (Ryzen AI Max+ 395) and faster than the current gen mainstream-class Intel desktop CPU (Core Ultra 5 250K Plus).

7-zip 26.01 file compression performance on various different CPUs.

The decompression performance also painted a good number.

7-zip 26.01 file decompression speed.

Albeit a much bigger gap against the Ryzen AI Max+ 395. Still respectable nonetheless.

7-zip 26.01 file decompression performance on various different CPUs.

Comparison against A Similar Mini AI Machine

It’s been a year or so that AMD released the Strix Halo (Ryzen AI Max+ 395 with integrated Radeon 8060S). I actually tested one but haven’t written any article about it (sorry 😥). Now when you see the raw specification, they’re kinda similar in stature.

DGX SparkStrix Halo
Memory128 GB 256-bit LPDDR5x unified128 GB 256-bit LPDDR5x shared
Memory Bandwidth273 GB/s256 GB/s
Smallest Floating Point SupportNVFP4, MXFP4BF16, FP16
Smallest Integer SupportINT4INT4

But it has this glaring difference: DGX Spark supports a smaller floating point data type.

Aside from the data type support, you can also look at the memory. DGX Spark uses unified memory while Strix Halo uses shared memory. This is different in nature.

With unified memory, you get a transparent memory addressing in which the CPU and GPU can access the same memory address, making it possible for the CPU or GPU-bound application to get more memory available on the pool on-demand. They can access the same 128 GB available memory in whole.

Strix Halo, uses shared memory in which you need to set a fixed amount of shared UMA (unified memory address) size available for the GPU. While the GPU can dynamically allocate the GPU memory to the available memory address, the fixed amount of shared UMA is locked for GPU and the CPU can only use the remaining memory pool. For example you can set the UMA size of 1 GB statically set to the GPU, then the

We’ll see the difference in performance later on.

LLM Performance

First, I need to talk about sidelining llama.cpp for vLLM. The reason is llama.cpp wouldn’t be representative for this machine LLM capability measurement. As I already presented above, it has a large unified memory capacity of 128 GB. With this much of a memory, you are not on the mercy of GGUF models with llama.cpp and you can just run the non-GGUF quantized model with an inference engine good enough to serve more requests.

Wait that explanation isn’t really helpful, right?

Let me just show you this model size comparison.

Here are unsloth/Qwen3.6-35B-A3B-GGUF models’ size.

With GGUF model, you can grab a lower quantization easily to scale with your memory capacity.

And this is unsloth/Qwen3.6-35B-A3B-NVFP4 model size.

This non-GGUF model only has one variant, which is 23 GB in size.

llama.cpp will allocate certain parts of the memory to be used as KV cache. This number is fixed based on the data type being used and the context length set up in the llama-server directive. It can’t grow or shrink to fit the available memory, nor set up on-the-fly when the inference server is running.

In this example, I used context length of 262144 tokens, K cache data type of q4_0, and V cache data type of q4_0. It used 1440 MB of the available memory for the context alone, independent from the memory needed to run the model.

Unlike llama.cpp, when you use an enterprise focused inference engine like vLLM, you can use the remaining available memory for more concurrency.

In this example, the remaining 67.5 GB after loading the data would be allocated as KV cache memory, which then translate to 13.3 of maximum concurrencies estimation.

Now when we put the comparison side-by-side, you can see that vLLM can serve more concurrent requests easily compared to llama.cpp, even with a similarly sized GGUF model. All the benchmarks were done with llama-benchy for easy benchmarking tool and consistency.

llama.cpp serve command line, model: unsloth/Qwen3.6-35B-A3B-GGUF, UD-Q4_K_XL quantization
Bash
llama-serve \
  --model unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --mmproj unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf --image-min-tokens 1024 \
  -ctk q8_0 -ctv q8_0 -c 262144 -ub 4096 -b 4096 \
  -fa 1 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --parallel 1 --threads 16 \
  --host 0.0.0.0 --port 8000 \
  --jinja
vLLM serve command line, model: unsloth/Qwen3.6-35B-A3B-NVFP4
Bash
podman run \
  -p 8000:8000 \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface:Z \
  docker.io/vllm/vllm-openai:cu130-nightly unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --trust-remote-code \
  --dtype float16 \
  --max-model-len -1 \
  --gpu-memory-utilization 0.8
llama-benchy benchmarking line for the llama.cpp server, model: unsloth/Qwen3.6-35B-A3B-GGUF, UD-Q4_K_XL quantization
Bash
llama-benchy \
  --base-url http://${HOST}:8000/v1 \
  --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --depth 0 8192 \
  --pp 2048 --tg 256 \
  --concurrency 1 2 4 \
  --no-results-on-fail
llama-benchy benchmarking line for the vLLM server, model: unsloth/Qwen3.6-35B-A3B-NVFP4
Bash
llama-benchy \
  --base-url http://${HOST}:8000/v1 \
  --model unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --depth 0 8192 \
  --pp 2048 --tg 256 \
  --concurrency 1 2 4 \
  --no-results-on-fail
The llama.cpp fell through in prompt processing performance when faced with more than 1 concurrent request.
Similar story with the token generation performance. Although not as dramatic as the prompt processing one.

Now that I’ve already put out my reasons, let’s continue to the vLLM benchmarks.

But wait, there’s more. I also put Radeon AI Pro R9700 as another data for comparison.

DGX SparkRadeon AI Pro R9700Strix Halo
Memory128 GB 256-bit LPDDR5x unified32 GB 256-bit GDDR6 dedicated128 GB 256-bit LPDDR5x shared
Memory Bandwidth273 GB/s644.6 GB/s256 GB/s
Smallest Floating Point SupportNVFP4, MXFP4FP8BF16, FP16
Smallest Integer SupportINT4INT4INT4

vLLM Container Image Version

For the vLLM deployment, I used these container images running on Podman for each machine, as I found them to be the most performant at the time of testing (18 May 2026).

Machine/GPUContainer Image
DGX Sparkdocker.io/vllm/vllm-openai:cu130-nightly
Radeon AI Pro R9700docker.io/rocm/vllm-dev:nightly
Strix Halodocker.io/kyuz0/vllm-therock-gfx1151:latest

Also here are the vLLM inference engine serving lines for each GPU:

vLLM serve command line for DGX Spark
Bash
podman run \
  -p 8000:8000 \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface:Z \
  docker.io/vllm/vllm-openai:cu130-nightly ${MODEL} \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --trust-remote-code \
  --dtype float16 \
  --max-model-len -1 \
  --gpu-memory-utilization 0.8
vLLM serve command line for Radeon AI Pro R9700
Bash
# Create distrobox
distrobox create vllm \
  --image docker.io/rocm/vllm-dev:nightly \
  -- \
  --device /dev/dri --device /dev/kfd \
  --group-add keep-groups --security-opt seccomp=unconfined

# Enter distrobox
distrobox enter vllm

# Run vllm serve inside the distrobox
vllm serve \
  ${MODEL} \
  --host 0.0.0.0 --port 8000 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --trust-remote-code \
  --dtype float16 \
  --max-model-len -1 \
  --gpu-memory-utilization 0.9 \
  --enforce-eager
vLLM serve command line for Strix Halo
Bash
# Create distrobox
distrobox create vllm-gfx1151 \
  --image docker.io/kyuz0/vllm-therock-gfx1151:latest \
  -- \
  --device /dev/dri --device /dev/kfd \
  --group-add keep-groups --security-opt seccomp=unconfined

# Enter distrobox
distrobox enter vllm-gfx1151

# Run vllm serve inside the distrobox
VLLM_ROCM_USE_AITER=1 vllm serve \
  ${MODEL} \
  --host 0.0.0.0 --port 8000 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --trust-remote-code \
  --dtype float16 \
  --max-model-len -1 \
  --gpu-memory-utilization 0.8
llama-benchy benchmarking for the vLLM servers
Bash
llama-benchy \
  --base-url http://${HOST}:8000/v1 \
  --model ${MODEL} \
  --depth 0 8192 16384 32768 \
  --pp 2048 --tg 256 \
  --concurrency 1 2 4 8 \
  --no-results-on-fail

vLLM Performance, Qwen/Qwen3.5-35B-A3B-GPTQ-Int4

First, I used an exact same model, Qwen/Qwen3.5-35B-A3B-GPTQ-Int4, which is Qwen3.5-35B-A3B model that has been quantized to int4 (4-bit integer). This way we can measure the performance without resorting to any hardware data type support advantage.

vLLM prompt processing performance using Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 from DGX Spark, R9700, and Strix Halo.

As we can see above, the Radeon AI Pro R9700 was able to beat the DGX Spark when the prompt has no meaningful context length. But, the DGX Spark was able to gain advantage, even so slightly, on deeper contexts. Unfortunately, the Strix Halo fumbled big with the performance was nowhere around those two.

vLLM token generation performance using Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 from DGX Spark, R9700, and Strix Halo.

Funnily enough, even with lower memory bandwidth (273 GB/s vs 644.6 GB/s), the DGX Spark was able to blow the Radeon AI Pro R9700 out of the water. Don’t even mention the Strix Halo, it failed to perform to even half of the DGX Spark performance, and even worse on deeper contexts.

vLLM Performance, Qwen/Qwen3.6-35B-A3B, Different Quantization

For this one I used same models with different quantization on the DGX Spark against both set of Radeons.

The GB10 in the DGX Spark is based on Nvidia Blackwell architecture which supports NVFP4. This support is shared with other Blackwell-based GPU such as RTX 50 Series, RTX PRO Blackwell Series, B100, B200, B300, etc. For that reason, I used unsloth/Qwen3.6-35B-A3B-NVFP4 to represent the lowest possible hardware quantization support.

As the Radeon AI Pro R9700 and Strix Halo only support 4-bit on the integer unit and not on the floating point unit, I used cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit. It uses activation-aware weight quantization technique from the MIT HAN Lab and mainly utilises int4 (4-bit integer).

vLLM prompt processing performance using Qwen/Qwen3.6-35B-A3B from DGX Spark, R9700, and Strix Halo.

This one brought a more interesting comparison between the DGX Spark and the Radeon AI Pro R9700. Somehow the DGX Spark was able to gain foothold on no context depth with 4 concurrent requests. Also the deeper contexts on the DGX Spark shined this time against the R9700. And yes, the Strix Halo performance was far behind those two.

vLLM token generation performance using Qwen/Qwen3.6-35B-A3B from DGX Spark, R9700, and Strix Halo.

The data portrays similar story. But this time, the DGX Spark was slightly faster and the R9700 was slightly slower, giving more performance delta between those two. Poor Strix Halo still struggled hard.

LLM Performance Conclusion

At the time of writing, DGX Spark can be found for around IDR170Mio (assuming the currency rate of IDR18k/USD), give or take, with bigger storage size would warrant higher price.

Now compare that to R9700 which can be found for around IDR30Mio. When you factor the total system cost of around IDR80Mio for a high-end system, yet yields LLM performance not far short of the DGX Spark, you would have to answer at least these 4 questions:

  1. Do you need a smaller footprint?
  2. Do you need lower heat and power envelope?
  3. Do you need a better software/model/documentation/support?
  4. Do you intend to setup a cluster?

If most of your answers are yes, I think the IDR170Mio of DGX Spark would worth the money compared to R9700 or Strix Halo.

I’m sorry Strix Halo. But with your current expected price for 128 GB variant of around IDR75Mio, I would rather choose between R9700 or DGX Spark.

Image Generation Performance

Things get more interesting here on image generation test. As I already wrote above regarding the NVFP4 support, you can expect that this mini machine would be able to run Nunchaku SVDQuant FP4 model like the RTX 5070 Ti we tested on this article. But there’s a big trouble challenge. Big big challenge.

The Nunchaku wheel binaries provided on Nunchaku Github release are only available for x86_64. Meanwhile this DGX Spark uses Arm processor inside.

Oof.

Oh wait. Turns out there is someone from Japan who was able to build and compile from source, so that Nunchaku wheel would run for aarch64 (Arm64)!

Yep. After following the direction in which I required to use Google Translate to decipher that article (shout out to tori29umai) and waited for several minutes of compilation process (trust me it was goddamn long lol, I’m used to faster CPUs), voila!

Those who do not know pain (of self-compiling) will never understand true peace (of simply installing).

I used the same model (Qwen Image Edit 2509), same workflow, and same input images for the comparison as the one used in this article.

TypeModel
GGUFBase: QuantStack/Qwen-Image-Edit-2509-Q4_K_M
LoRA: lightx2v/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16
Nunchaku SVDQuant NVFP4nunchaku-ai/nunchaku-qwen-image-edit-2509/svdq-fp4_r32-qwen-image-edit-2509-lightningv2.0-4steps

First, I would like to show you how the DGX Spark stands against the R9700 and the Strix Halo, using GGUF model which can be run across those three.

DGX Spark – Qwen Image Edit 2509 GGUF run screenshot

Radeon AI Pro R9700 – Qwen Image Edit 2509 GGUF run screenshot
Strix Halo – Qwen Image Edit 2509 GGUF run screenshot

Qwen Image Edit 2509 GGUF performance comparison.

One interesting thing is, the Radeon AI Pro R9700 is now much faster than the last time I tested.

52.17s then to 36.87s now translates to around 30% time shaving! Kudos to AMD!

The Strix Halo is around 4x slower than the other two. Also faster than the last time I tried which requires around 188s to complete the task.

This was the last time I ran Qwen Image Edit 2509 GGUF on the Strix Halo.

Now, let’s get to the staple of Nunchaku SVDQuant FP4 variant.

DGX Spark – Qwen Image Edit 2509 SVDQuant FP4 run screenshot

Qwen Image Edit 2509 complete performance comparison.

The DGX Spark finished the task and shaved around 38% of required time to complete using SVDQuant FP4 model variant. Turns out having a 4-bit floating point unit still helps even with a smaller chip and lower power.

Desktop PC Experience

If you want to switch the experience from being a mini supercomputer to a mere mini PC, it handles the job well. Well enough that you can do browsing, but nothing special from the performance standpoint.

Reasonable.
Comparison to other systems, courtesy of NotebookCheck.net. Apparently faster than Strix Halo represented by the Asus ProArt PX13 there.

For multimedia purposes, it supports a wide range of codec on its encoder and decoder.

Encoder codec support: YES YES YES YES YES YES…
…YES YES YES YES YES
Decoder codec support: ALSO GIVE ME AS MANY YES…
…AS POSSIBLE

And my test showed that it is possible to use the NVENC (Nvidia encoder) to do some video rendering with the state of the art AV1 codec.

NVENC AV1 is available!
AV1 video encoding demonstration using Kdenlive.
AV1 video encoding demonstration using Handbrake.

Gaming

Funnily enough, you can also game on DGX Spark. Despite of having a full-fledged Nvidia GPU with shader core, raytrace core, and tensor core, you won’t be expected to game on this as well as a normal gaming PC. The reason is most of the game available there are developed for:

  1. Windows (or DirectX API, to be precise)
  2. x86 CPU

Well, this system is running on:

  1. Linux (Vulkan API)
  2. Arm CPU

Now think about it. In order for your ordinary game to work, it needs to translate/emulate both the graphics API and the CPU ISA, before it hits the underlying low level instructions on the GPU and the CPU. The translation layers require certain computational cost in order for it to run.

For example, certain games require you to have an x86 CPU with AVX2 instruction extension. But there’s no such thing on Arm, and no such thing on GB10 CPU as well.

GB10 CPU flags. Do you see any AVX/AVX2/AVX512 anywhere?

Thanks to the people from the Box86/Box64 project, we actually have an x86 to Arm emulator. Kudos to them, they’re also managed to emulate AVX(2)! Therefore, this Steam for Arm64 on Ubuntu was born.

Steam snap for Arm64

And yes, with the magic of Box64 and Proton and VKD3D and many many more open source projects, I managed to play Clair Obscur: Expedition 33!

Expedition 33 in all of its glory.

Unfortunately I wasn’t able to make MangoHud or Steam overlay to work, so there’s no framerate reading available. It was playable, not as smooth as my own PC, but respectable nonetheless for a machine which is not intended for gaming.

Power Consumption

DGX Spark only exposes the power draw data from the GPU side. I tried probing the available sensors exposed to the OS to find a way to read the CPU or the SoC power draw using various tools like lm-sensors, siomon, and Prometheus node_exporter, to no avail. In the end, I only had limited time doing the power draw test. Regardless I can show you some interesting data, using both the GPU power draw sensor and wall outlet watt meter.

Video encoding GPU power draw of around 15-17W.
Image generation GPU power draw of around 85W, with idle GPU power draw of around 10-11W.
Image generation wall outlet power draw of around 160W.
Expedition 33 gaming GPU power draw of around 45W.
Expedition 33 gaming wall outlet power draw of around 126-130W.

Noise

Every implementation of the DGX Spark platform from various Nvidia partners brings their own cooling solution. As for this test, it is specific to MSI EdgeXpert and not representative to all DGX Spark solutions available.

I would describe the noise as a typical gaming laptop whisper which can get louder under load. Even at the loudest it’s still pretty reasonable and you can have this system on your desk no problem.

Idle noise.
Under load noise.

Verdict

What can we grab from all those tests?

The fact that this has no AI moniker in its name yet excels in pretty much all of my AI tests, impressed me by very much.

If you’re looking for an edge device, with excellent support, software, and matured CUDA platform, maybe this device is for you. The problem would be the price. My estimation of IDR170Mio as per 28th May 2026 is based on the uncertainty of the currency exchange rate, the memory/storage chip shortage, and the freight cargo price hike caused by the Middle East conflict. It can probably get more expensive in the future, with no option of waiting until the price drops.

If you’re looking for alternatives, I don’t feel there’s any non-enterprise system or GPU that can give 128 GB of total available memory with this level of performance, especially around vLLM concurrent requests and deep context which are required for agentic AI of today. Not to mention the possibility of stacking multiple DGX Spark and setting up tensor parallelism to run a bigger model or to get more performance.

Also pls don’t buy it for a gaming PC. Save your money and buy any typical desktop or laptop instead.

The post Our Experience with MSI EdgeXpert – Nvidia DGX Spark GB10 appeared first on Efison Lisan Teknologi.

]]> Our Experience with Asus AMD Radeon AI Pro R9700 Turbo https://efisonlt.com/our-experience-with-asus-amd-radeon-ai-pro-r9700-turbo/?utm_source=rss&utm_medium=rss&utm_campaign=our-experience-with-asus-amd-radeon-ai-pro-r9700-turbo Sat, 11 Oct 2025 13:13:19 +0000 https://efisonlt.com/?p=1880 2025, and AI. What’s not to love? Again, if somebody were to sell a rendang and they state that it was created using AI, I bet venture capitals would clap and circle like vultures. Okay enough yapping. Now we are talking about a damn GPU. A tool to run real AI. Introducing, AMD Radeon AI… Read More »Our Experience with Asus AMD Radeon AI Pro R9700 Turbo

The post Our Experience with Asus AMD Radeon AI Pro R9700 Turbo appeared first on Efison Lisan Teknologi.

]]> 2025, and AI.

What’s not to love?

Again, if somebody were to sell a rendang and they state that it was created using AI, I bet venture capitals would clap and circle like vultures.

Okay enough yapping. Now we are talking about a damn GPU. A tool to run real AI.

Introducing, AMD Radeon AI Pro R9700

In their wisdom, calling a prosumer/workstation GPU with a “pro” moniker doesn’t quite cut it anymore. It now has AI in its name. Why, do you ask?

Because apparently it has 32GB worth of VRAM and it supports another numerical precision (FP8). The VRAM is quite big, we reckon. Still not as big as the last gen Pro (no AI) W7900, but with all its new-ness, AI-ness, and goodness, it’s still pretty nice.

Radeon AI Pro R9700Radeon Pro W7900
ArchitectureAMD RDNA 4AMD RDNA 3
Memory size32 GB48 GB
Memory bandwidth644.6 GB/s864 GB/s
Memory ECC supportYes (Linux only)*Yes
Peak FP32 (vector) performance47.8 TFLOPS61.3 TFLOPS
Peak FP16 (vector) performance95.7 TFLOPS123 TFLOPS
Peak FP16 (matrix) performance191 TFLOPS123 TFLOPS
Peak FP8 (matrix) performance383 TFLOPSN/A
Peak INT8 (matrix) performance383 TOPS123 TOPS

Personally we’re pretty confused about why the memory ECC support on R9700 is stated as Linux only but as we’re mostly using Linux as our test platform, no complaint there.

Even with smaller memory size, it is pretty beefy in terms of performance. Especially in theoretical matrix performance. Hence the AI namesake. Unfortunately we have no W7900 or any RX 7900 XTX for real-world comparison purpose, but we do have an RTX 5070 Ti which apparently has a similar profile.

Radeon AI Pro R9700Geforce RTX 5070 Ti
Process technologyTSMC N4PTSMC 4N
Die size357 mm²378 mm²
Memory size32 GB GDDR616 GB GDDR7
Memory interface256-bit256-bit
Memory bandwidth644.6 GB/s896 GB/s
Total board power300 W300 W

The Test Setup

ComponentSpecification
CPUIntel Core i7-12700K
@5.0 GHz
MotherboardASRock Z690 PG Velocita
Memory4*24 GB Klevv Cras V RGB
@DDR5-5600
Storage2TB MSI Spatium M480
PSU1000W 1stPlayer NGDP Gold

For the benchmark, we tested LLM using llama.cpp and image generation using Qwen Image Edit 2509.

llama.cpp Benchmark Setup

We used llama.cpp build d2ee056e1 (6713) and compiled the CPU backend using Intel oneAPI compiler 2025.2.1 against external BLAS library which is Intel oneAPI MKL 2025.2. Why, do you ask? Because it yields faster performance compared to mere GNU compiler 15.2.1 with no BLAS.

We tested using unsloth/gpt-oss-120b-Q4_K_M model and .(7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(up|down|gate)_exps. MoE layers which are then being put to system RAM for CPU offload processing.

(In token/s. Higher is better)GNU compiler 15.2.1
no BLAS		oneAPI compiler 2025.2.1
BLAS=oneAPI MKL 2025.2
Prompt processing (512 tokens)180.65 ± 1.74182.39 ± 1.60
Text generation (256 tokens)21.85 ± 0.8232.19 ± 0.04
Compilation steps for GNU compiler 15.2.1, no BLAS
Bash
mkdir build_no-blas-gcc_vulkan && cd build_no-blas-gcc_vulkan
cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -j
Compilation steps for oneAPI compiler 2025.2.1, BLAS=oneAPI MKL 2025.2
Bash
mkdir build_mkl-ilp64-icx_vulkan && build_mkl-ilp64-icx_vulkan
cmake -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -j
Run output
Bash
GGML_VULKAN_DEVICE=0 ./build_no-blas-gcc_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan     |  99 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        180.65 ± 1.74 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan     |  99 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         21.85 ± 0.82 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        182.39 ± 1.60 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         32.19 ± 0.04 |

build: d2ee056e1 (6713)

For the GPU backend, we used both Vulkan and HIP (ROCm 7) which we won’t discuss much here to prevent spoilers.

We refrained on using ROCm 6.4.x as ROCm 7.0.x is now performing much better on this GPU (or probably all AMD RDNA4 GPUs in general). If you haven’t heard already, AMD have just released their newest ROCm 7 on September 16, 2025. We did a quick comparison in terms of llama.cpp performance against ROCm 6.4.3 which can be seen on this guy’s Facebook post.

As for the RTX 5070 Ti, we compiled llama.cpp GPU backend against CUDA without GGML_CUDA_FORCE_CUBLAS.

So for references, here are the compilation lines used for various configurations stated above:

Compilation line for llama.cpp ROCm
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_HIP=ON -DGPU_TARGETS=gfx1201
Compilation line for llama.cpp Vulkan
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1
Compilation line for llama.cpp CUDA
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_CUDA=ON

We chose 2 models to be used for testing:

Qwen Image Edit 2509 Benchmark Setup (and rambling about PyTorch for ROCm on Windows situation)

This one is a bit different because it has PyTorch dependencies and it’s not that simple. Historically, AMD has been neglecting (or unable to make?) Windows PyTorch package for their GPUs. With the Windows Preview Edition 25.20.01.14 driver, they finally support PyTorch on Windows for Radeons. Yay!

Albeit the limited roster of supported GPUs…

Funnily enough, they only listed Windows 11 as the compatible OS. Lo and behold, we managed to use it on Windows 10.

Enough with the rambling, now let’s get onto the setup.

We did the tests on 2 different OSes:

  • Aurora Linux 42 based on Fedora Kinoite
  • Windows 10 Pro

For the PyTorch, we tested different combinations:

PyTorch 2.x for ROCm 6.4.xPyTorch 2.x for ROCm 7.0.x
Aurora Linux 42✘✔
Windows 10 Pro✔✔

The Qwen Image Edit 2509 models used here were QuantStack/Qwen-Image-Edit-2509-Q4_K_M. We also used lightx2v/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16 LoRA for performance reason. As for the UI and additional package, we used ComfyUI v0.3.64. Not to forgot to mention city96/ComfyUI-GGUF for GGUF models compatibility.

Additional notes about Qwen Image Edit 2509 test setup and deployment

As for the inputs, we used these images:

And these sentences for the prompt:

Test Results: llama.cpp

We talked a bit about llama.cpp LLM MoE layer offload to CPU above. But in short, CPU offloading is done to make sure you can run a big model without the big performance hit caused by memory spill from VRAM to system RAM.

So, if you run a big model on GPU, and the VRAM is smaller than the model, it will still run. But it will run terribly because now the GPU needs to access the data from RAM in which the available VRAM can’t contain, while the CPU is doing nothing to help the processing.

unsloth/gpt-oss-120b-Q4_K_M has the size of 58.45 GB, while the available VRAM is only around 32 GB before substracting other necessary services for the operating system, which is now spilling around 26-27 GB to system RAM.

Now see the difference with CPU offloading.

With .(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(up|down)_exps. layers being offloaded to CPU and put to the system RAM, now the CPU can help with the processing while preventing the layers loaded to VRAM to spill over.

Probably this article is not the best at explaining how the MoE layers work or the way they’re offloaded. You can read a little bit more technical stuffs starting from reading this guide, or maybe a little bit of Google-fu. Sorry.

Now, let’s see the performance difference:

(In token/s. Higher is better)Without CPU offloadingWith CPU offloading
Prompt processing (512 tokens)120.48 ± 4.06215.93 ± 1.42
Text generation (256 tokens)11.21 ± 0.0234.97 ± 0.15
Run output
Bash
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        120.48 ± 4.06 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |         11.21 ± 0.02 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        215.93 ± 1.42 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         34.97 ± 0.15 |

build: d2ee056e1 (6713)

Let’s continue with the test results.

Here we have different configs of layers for CPU offloading. One is to keep the VRAM usage to under 16 GB, and then maximize the possible 32 GB. The configuration details can be seen below:

  • RTX 5070 Ti with just under 16 GB VRAM load, CUDA backend
  • RTX 5070 Ti with just under 16 GB VRAM load, Vulkan backend
  • R9700 with just under 16 GB VRAM load, ROCm 7 backend
  • R9700 with just under 16 GB VRAM load, Vulkan backend
  • R9700 with just under 32 GB VRAM load, ROCm 7 backend
  • R9700 with just under 32 GB VRAM load, Vulkan backend

Here are the results using unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS:

(In token/s. Higher is better)5070 Ti, 16G
CUDA		5070 Ti, 16G
Vulkan		R9700, 16G
ROCm 7		R9700, 16G
Vulkan		R9700, 32G
ROCm 7		R9700, 32G
Vulkan
Prompt processing (512 tokens)3723.33 ± 50.092739.83 ± 30.37746.77 ± 3.451236.97 ± 9.19797.92 ± 3.611665.47 ± 5.95
Text generation (256 tokens)137.34 ± 0.69138.41 ± 0.7988.98 ± 0.16105.35 ± 0.75100.05 ± 0.08122.63 ± 0.52

Here are the results using unsloth/gpt-oss-120b-Q4_K_M:

(In token/s. Higher is better)5070 Ti, 16G
CUDA		5070 Ti, 16G
Vulkan		R9700, 16G
ROCm 7		R9700, 16G
Vulkan		R9700, 32G
ROCm 7		R9700, 32G
Vulkan
Prompt processing (512 tokens)370.32 ± 4.22206.81 ± 3.52188.32 ± 4.82169.56 ± 2.60251.93 ± 6.61230.01 ± 2.78
Text generation (256 tokens)40.24 ± 0.1337.90 ± 0.4232.59 ± 0.0131.49 ± 0.0438.73 ± 0.0836.22 ± 0.03
RTX 5070 Ti 16G run output
Bash
./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |      3723.33 ± 50.09 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        137.34 ± 0.69 |

build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |      2739.83 ± 30.37 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        138.41 ± 0.79 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        370.32 ± 4.22 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         40.24 ± 0.13 |

build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        206.81 ± 3.52 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         37.90 ± 0.42 |

build: d2ee056e1 (6713)
R9700 16G run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |        746.77 ± 3.45 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |         88.98 ± 0.16 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |       1236.97 ± 9.19 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        105.35 ± 0.75 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        188.32 ± 4.82 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         32.59 ± 0.01 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        169.56 ± 2.60 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         31.49 ± 0.04 |

build: d2ee056e1 (6713)
R9700 32G run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        797.92 ± 3.61 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        100.05 ± 0.08 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |       1665.47 ± 5.95 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        122.63 ± 0.52 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        251.93 ± 6.61 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         38.73 ± 0.08 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        230.01 ± 2.78 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         36.22 ± 0.03 |

build: d2ee056e1 (6713)

Even though it has twice the VRAM, with the MoE layers CPU offloading strategy, turns out it still can’t quite compete with the RTX 5070 Ti. This might be caused by the large memory bandwidth gap between both (644.6 GB/s vs 896 GB/s, 39% difference).

Now, when you see on unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS, it has this big performance delta between ROCm 7 backend and Vulkan backend. I wonder why a bunch of open-source developers from Vulkan/SPIR-V/Kompute project can outperform engineers who are paid to make their stuffs performant… Although, it did better on unsloth/gpt-oss-120b-Q4_K_M by a tiny margin.

Also, turns out having a bigger VRAM still can’t quite defeat the smaller VRAM, if your bandwidth is too slow, and your software isn’t up to par.

It’s not about the size. It’s about how you use it.

Test Results: Qwen Image Edit 2509

Sometimes we question ourselves why do we like to suffer.

And this test is no different.

We (or me, personally), wonder if being a goose farmer is a better choice for living a happy life.

First, let us tell you that running this test for the first run, is not a happy feat. It takes quite a while for the text encoder and VAE to load and process our prompt and image inputs.

On a bad day, it can take more than 7 mins of time to generate this for the first run.

UPDATE: IT DOESN’T HAVE TO BE 7 MINUTES

PyTorch for ROCm apparently has a bug in which the VAE stage is extremely slow.

The solution? ComfyUI recently pushed a workaround. Which is to disable cuDNN back-end (?).

Now the first generation isn’t that painful anymore.

Original article continues below.

Here, we only put the generation time after text encoder and VAE had been loaded and then run it in a batch of 5. Then we averaged the time needed to generate the same prompt with the same inputs.

Now, let’s get onto the results:

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux		R9700
ROCm 7.0.x
Linux		R9700
ROCm 6.4.x
Windows		R9700
ROCm 7.0.x
Windows
Results29.38452.1769.26262.59
RTX 5070 Ti CUDA 12.9 Linux test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-GGUF.png

R9700 ROCm 7.0.x Linux test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF.png

R9700 ROCm 6.4.x Windows test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF.png

R9700 ROCm 7.0.x Windows test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF-gfx120x-nightly.png

Oops.

Almost twice faster.

Also you can make a case that PyTorch for ROCm on Windows is slower. Almost 20% slower than on Linux. Although this is only on one use case that we tested and we didn’t confirm with different various use cases.

And to make it worse, RTX 5070 Ti does have a trick up its sleeve.

Introducing Nunchaku SVDQuant. An inference engine so fast it cut the inference time of Qwen Image Edit 2509 almost in half even versus already quantized Q4_K_M model.

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux
Nunchaku FP4 r32		RTX 5070 Ti
CUDA 12.9
Linux
Q4_K_M GGUF
Results15.19829.384
RTX 5070 Ti CUDA 12.9 Linux, Nunchaku FP4 r32 model test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-Nunchaku-r32-CPU-offload.png

Overclocking

Real men do OC.

Or men with too much times in hands.

We managed to overclock this card using LACT. No, we didn’t test overclocking on Windows. Penguins FTW!

We found out that 2800 MHz was the ideal maximum VRAM clock to keep it from crashing doing various workloads. Still a healthy 282 MHz increase from the default 2518 MHz. Which translates to 11.2% more memory bandwidth.

The maximum power usage limit also increased to 330 W from 300 W (10% increase). And maybe you wonder what does GPU voltage offset about? Why is it being lowered, right?

Some of you might have heard that overclocking the GPU clock on RDNA4 is done by shifting the voltage/frequency curve by setting the voltage offset to a lower value, so the GPU would be tricked and boosts to higher frequency.

Too difficult to understand? Let us show you a video from our friend Alva Jonathan who did an excellent job explaining overclocking on another RDNA4 GPU which is a Radeon RX 9070.

Here are the performance results to show you the gain from the overclocking attempt:

llama.cpp unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS (Overclocked)

(In token/s. Higher is better)5070 Ti, 16G
CUDA		R9700, 32G
Vulkan
Overclocked		R9700, 32G
Vulkan
Stock default
Prompt processing (512 tokens)3723.33 ± 50.091810.00 ± 11.361665.47 ± 5.95
Text generation (256 tokens)137.34 ± 0.69131.09 ± 0.37122.63 ± 0.52

llama.cpp unsloth/gpt-oss-120b-Q4_K_M (Overclocked)

(In token/s. Higher is better)5070 Ti, 16G
CUDA		R9700, 32G
ROCm 7
Overclocked		R9700, 32G
ROCm 7
Stock default
Prompt processing (512 tokens)370.32 ± 4.22254.46 ± 4.62251.93 ± 6.61
Text generation (256 tokens)40.24 ± 0.1339.72 ± 0.0438.73 ± 0.08
Overclocked R9700 llama.cpp run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        834.27 ± 3.99 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        105.42 ± 0.12 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |      1810.00 ± 11.36 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        131.09 ± 0.37 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        254.46 ± 4.62 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         39.72 ± 0.04 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        236.58 ± 2.89 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         37.41 ± 0.04 |

build: d2ee056e1 (6713)

As we can see, the increase is bigger when the model is smaller, as bigger portion of layers fit on the VRAM hence the performance gain from VRAM overclocking. Still a tad slower than RTX 5070 Ti, but we’ll grab what we can.

Qwen Image Edit 2509 Q4_K_M GGUF (Overclocked)

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux		R9700
ROCm 7.0.x
Linux
Overclocked		R9700
ROCm 7.0.x
Linux
Stock default
Results29.38448.62852.17
Overclocked R9700 run screenshot

https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF-OC.png

7.28% faster. Not bad for a free performance gain.

Verdict

So, what do we think?

Bigger VRAM doesn’t always translate to a bigger performance. Especially in our tests which are very inference heavy. This doesn’t mean this card is DoA or something (pls don’t be, we need alternatives to the leather jacketed overlord).

There are many good takeaways. Such as AMD GPU software team are now finally getting their stuffs together. PyTorch for ROCm on Windows was one of the biggest to-do list. ROCm 7 is being faster than ROCm 6 (at least on RDNA4).

They also managed to lower the size of ROCm Docker image and PyTorch package.

Unfortunately, turns out it was due to they chose to drop RDNA2 from their supported GPUs. Sad.

AMD Radeon ROCm 7 supported GPUs

We still think there would be a better value for a bigger VRAM for different use cases. Probably in LLM finetuning, in 3D modelling/rendering, which we presume wouldn’t be easier to work around with CPU offload the way we tested MoE LLM inference above.

The post Our Experience with Asus AMD Radeon AI Pro R9700 Turbo appeared first on Efison Lisan Teknologi.

]]> Menggunakan ALELEON Supercomputer, Dr. Fajar Inggit Pambudi, M.Sc. Menyelesaikan Penelitiannya Tentang Kerangka Logam-Organik Berbasis Zirkonium! https://efisonlt.com/menggunakan-aleleon-supercomputer-dr-fajar-inggit-pambudi-m-sc-menyelesaikan-penelitiannya-tentang-kerangka-logam-organik-berbasis-zirkonium/?utm_source=rss&utm_medium=rss&utm_campaign=menggunakan-aleleon-supercomputer-dr-fajar-inggit-pambudi-m-sc-menyelesaikan-penelitiannya-tentang-kerangka-logam-organik-berbasis-zirkonium Mon, 03 Jul 2023 08:50:28 +0000 https://efisonlt.com/?p=1851 Pada Juni 2023 lalu, Dr. Fajar Inggit Pambudi, M.Sc. berhasil mempublikasikan penelitiannya ke jurnal Materials Today Communications sebagai pemenang EUREKA! periode 2. Bekerjasama dengan 2 peneliti lainnya yaitu Nadiyah Sekar Pratiwi dan Ukhti Chusnawati, mereka mempublikasikan paper penelitian yang berjudul “First-principle study on the lattice-directed missing linker defect in zirconium based metal-organic framework (MOF-801): Electronic… Read More »Menggunakan ALELEON Supercomputer, Dr. Fajar Inggit Pambudi, M.Sc. Menyelesaikan Penelitiannya Tentang Kerangka Logam-Organik Berbasis Zirkonium!

The post Menggunakan ALELEON Supercomputer, Dr. Fajar Inggit Pambudi, M.Sc. Menyelesaikan Penelitiannya Tentang Kerangka Logam-Organik Berbasis Zirkonium! appeared first on Efison Lisan Teknologi.

]]> Pada Juni 2023 lalu, Dr. Fajar Inggit Pambudi, M.Sc. berhasil mempublikasikan penelitiannya ke jurnal Materials Today Communications sebagai pemenang EUREKA! periode 2.

Bekerjasama dengan 2 peneliti lainnya yaitu Nadiyah Sekar Pratiwi dan Ukhti Chusnawati, mereka mempublikasikan paper penelitian yang berjudul “First-principle study on the lattice-directed missing linker defect in zirconium based metal-organic framework (MOF-801): Electronic properties and interaction with hydrogen”.

Penelitian ini bertujuan untuk mengetahui efek dari cacat yang diakibatkan oleh sambungan yang terputus serta prospek ke depan dari MOF-801 (kerangka logam-organik berbasis Zirkonium) untuk penyimpanan gas hidrogen.

Lattice-directed missing linker defect in MOF-801

Selamat kepada Dr. Fajar Inggit Pambudi, M.Sc. dan tim. Semoga perkembangan ilmu pengetahuan dan teknologi di Indonesia semakin berkembang bersama EFISON dan ALELEON Supercomputer!

Tautan paper:
www.sciencedirect.com/science/article/abs/pii/S235249282300658X

The post Menggunakan ALELEON Supercomputer, Dr. Fajar Inggit Pambudi, M.Sc. Menyelesaikan Penelitiannya Tentang Kerangka Logam-Organik Berbasis Zirkonium! appeared first on Efison Lisan Teknologi.

]]> Berkat ALELEON Supercomputer, Dr. Sc. Sholihun, S.Si., M.Sc. Berhasil Menemukan Kandidat Nanomaterial untuk Mengubah Panas Menjadi Listrik! https://efisonlt.com/aleleon-supercomputerberhasil-menemukan-kandidat-nanomaterial/?utm_source=rss&utm_medium=rss&utm_campaign=aleleon-supercomputerberhasil-menemukan-kandidat-nanomaterial Tue, 05 Jul 2022 11:51:25 +0000 https://efisonlt.com/?p=1805 Dr. Sc. Sholihun, S.Si., M.Sc. sebagai pemenang EUREKA! periode 1 berhasil mempublikasikan hasil penelitiannya ke jurnal internasional IOP Science! Dibantu dengan 3 peneliti lainnya yaitu P. Lubis, N. Amalia, dan S. A. Wella, penelitian yang berjudul “Thermoelectric properties of monolayer and bilayer buckled XTe (X = Ge, Sn, and Pb)” ini bertujuan untuk mencari kandidat… Read More »Berkat ALELEON Supercomputer, Dr. Sc. Sholihun, S.Si., M.Sc. Berhasil Menemukan Kandidat Nanomaterial untuk Mengubah Panas Menjadi Listrik!

The post Berkat ALELEON Supercomputer, Dr. Sc. Sholihun, S.Si., M.Sc. Berhasil Menemukan Kandidat Nanomaterial untuk Mengubah Panas Menjadi Listrik! appeared first on Efison Lisan Teknologi.

]]> Dr. Sc. Sholihun, S.Si., M.Sc. sebagai pemenang EUREKA! periode 1 berhasil mempublikasikan hasil penelitiannya ke jurnal internasional IOP Science!

Dibantu dengan 3 peneliti lainnya yaitu P. Lubis, N. Amalia, dan S. A. Wella, penelitian yang berjudul “Thermoelectric properties of monolayer and bilayer buckled XTe (X = Ge, Sn, and Pb)” ini bertujuan untuk mencari kandidat nanomaterial yang dapat mengubah energi panas menjadi energi listrik secara langsung. Seluruh pengujian ini dilakukan dengan komputasi canggih yang dibantu oleh ALELEON Supercomputer bertenaga AMD EPYC!

Selamat atas publikasi paper penelitiannya. Semoga ilmu pengetahuan dan teknologi di Indonesia semakin berkembang dengan hadirnya EFISON dan ALELEON Supercomputer!

Tautan paper
iopscience.iop.org/article/10.1088/2043-6262/ac7322









The post Berkat ALELEON Supercomputer, Dr. Sc. Sholihun, S.Si., M.Sc. Berhasil Menemukan Kandidat Nanomaterial untuk Mengubah Panas Menjadi Listrik! appeared first on Efison Lisan Teknologi.

]]> Review ASUS ESC4000A-E10, server 2U dengan AMD EPYC dan Nvidia Tesla untuk HPC https://efisonlt.com/review-asus-esc4000a-e10/?utm_source=rss&utm_medium=rss&utm_campaign=review-asus-esc4000a-e10 Mon, 23 Nov 2020 10:16:48 +0000 https://efisonlt.com/?p=889 ASUS telah dikenal oleh umum sebagai produsen berbagai produk konsumer berkualitas seperti gadget, laptop, maupun komponen PC. Mengikuti perkembangan permintaan pasar akan komputasi kelas enterprise, saat ini ASUS mulai menapaki pasar server dengan lebih serius. Kali ini EFISON kedatangan server ASUS ESC4000A-E10 yang ditenagai CPU AMD EPYC Rome dengan arsitektur AMD Zen 2 dan GPU… Read More »Review ASUS ESC4000A-E10, server 2U dengan AMD EPYC dan Nvidia Tesla untuk HPC

The post Review ASUS ESC4000A-E10, server 2U dengan AMD EPYC dan Nvidia Tesla untuk HPC appeared first on Efison Lisan Teknologi.

]]> ASUS telah dikenal oleh umum sebagai produsen berbagai produk konsumer berkualitas seperti gadget, laptop, maupun komponen PC. Mengikuti perkembangan permintaan pasar akan komputasi kelas enterprise, saat ini ASUS mulai menapaki pasar server dengan lebih serius. Kali ini EFISON kedatangan server ASUS ESC4000A-E10 yang ditenagai CPU AMD EPYC Rome dengan arsitektur AMD Zen 2 dan GPU NVIDIA dengan arsitektur NVIDIA Turing.

ASUS ESC4000A-E10
ASUS ESC4000A-E10

Spesifikasi

Tipe Model/Spesifikasi
Processor / System Bus 1 x Socket SP3 (LGA 4094)
AMD EPYC™ 7002 Series
Memory Total Slots : 8 (8-channel)
Capacity : Maximum up to 2048GB RDIMM
Memory Type :
DDR4 3200 RDIMM
DDR4 3200 LRDIMM
DDR4 3200 LR-DIMM 3DS
Memory Size :
256GB, 128GB, 64GB, 32GB, 16GB
* Refer to support page for more information
Expansion Slots Rear:
– 4 x PCIe x16 slots (Gen4 x16 link, FH,FL) for dual-slot GPU cards or 8 x PCIe x16 slots (Gen4 x8 link, FH,FL) for single-slot GPU cards
– 2 x PCIe x16 slots (Gen4 x16 link, LP,HL)

Front:
– SKU-1 (default)
1 x PCIe x8 slot (Gen4 x8 link, LP,HL)

– SKU-2 (by request)
1 x PCIe x8 slot (Gen4 x8 link, LP,HL) or
1 x OCP3.0 slot (Gen4 x8 link) by switching cables

– SKU-3 (by request)
1 x PCIe x8 slot (Gen4 x8 link, LP,HL) or
2 x M.2 socket (Gen4 x4 link, up to 22110 module) by switching cables
Storage SATA Controller :
CPU Integrated
1 x M.2 connector(2242/2260/2280/22110) PCIe mode (PCIe Gen4 x4 link 22110/2280/2260)
Optional kits Controller :
ASUS PIKE II 3008-8i 8-port SAS 12G RAID card
ASUS PIKE II 3108-8i 8-port SAS 12G HW RAID card
Drive Bays 8 x 3.5″ or 2.5″ Hot-swap Storage Bays
(Backplane Supports 4 x SATA/SAS + 4 x SATA/SAS/NVMe Devices)

*default setting supports 2 x NVMe devices.
Networking 1 x Dual Port Intel I350-AM2 Gigabit LAN controller + 1 x Mgmt LAN
Graphic
Aspeed AST2500 with 64MB VRAM
Front I/O Ports 1 x Q-code/port-80 LED
4 x USB 3.2 Gen1 ports
Rear I/O Ports 2 x USB 3.2 Gen1 port
1 x VGA port
1 x Management port (RJ45)
2 x Gigabit LAN ports (RJ45)
Switch/LED Rear Switch/LED:
1 x Power switch/LED
1 x Location LED
1 x Message LED
1 x HDD Access LED

Front Switch/LED:
1 x Power switch/LED
1 x Location switch/LED
2 x LAN LED
1 x Message LED
1 x HDD LED
OS Support Please find the latest OS support from https://www.asus.com/event/Server/OS_support_list/OS.html
Management Solution ASUS Control Center
On-Board ASMB9-iKVM for KVM-over-IP
Regulatory Compliance BSMI, CE, RCM, FCC(Class A)
Dimensions 800mm x 440mm x 88.9mm (2U)
31.50″ x 17.22″ x 3.5″
Form Factor 2U
Power Supply 1+1 Redundant 1600W 80 PLUS Platinum Power Supply
Rating: 100-127Vac/200-240Vac,12.9A/9.5A,50-60Hz

1+1 Redundant 2200W 80 PLUS Platinum Power Supply
Rating: 100-127Vac/200-240Vac,14A/12.6A,47-63Hz
Environment Operation temperature: 10℃ ~ 35℃ / Non operation temperature: -40℃ ~ 70℃
Non operation humidity: 20% ~ 90% ( Non condensing)
Note
*Users need to remove Slimline cables from the PCIe riser board, and re-connect the cables to the backplane for total 4 x NVMe devices.

CPU yang datang ke lab EFISON untuk pengujian ASUS ESC4000A-E10 adalah AMD EPYC 7502P. EPYC 7502P memiliki 32 core dengan SMT yang menjadikannya 64 thread, boost clock hingga 3.35 GHz, dan L3 cache sebesar 128 MB. Dari sisi dukungan ia mendukung memori DDR4 ECC 8-channel dan PCIe 4.0 hingga 128-lanes. CPU ini merupakan salah satu dari jajaran CPU AMD EPYC Rome yang menggunakan mikroarsitektur Zen 2 dengan fabrikasi 7 nm TSMC.

AMD EPYC 7502P dengan memori 8-channel
AMD EPYC 7502P dengan memori 8-channel

Dari sisi GPU, EFISON juga diberi kesempatan untuk mencoba sebuah Nvidia Tesla T4. GPU ini menggunakan mikroarsitektur Turing dengan fabrikasi 12 nm. Tesla T4 menggunakan chip TU104 dengan 40 SM dan 2560 CUDA core. Ia memiliki ukuran single slot sehingga dapat dengan mudah dipasang di rack server 1U maupun 2U. Dari segi konsumsi daya ia hanya menggunakan daya dari slot PCIe yang tidak sampai 75 W. GPU ini dapat dipasang hingga 8 buah di server ASUS ESC4000A-E10 melalui slot PCIe di sebelah kanan dan kiri unit server.

Unit Nvidia Tesla T4
Unit Nvidia Tesla T4

8 slot PCIe x16 yang ada di sebelah kiri dan kanan
8 slot PCIe x16 yang ada di sebelah kiri dan kanan

ASUS ESC4000A-E10 mendukung hingga 8 buah 3.5″/2.5″ drive bay dengan konfigurasi maksimum 4 SATA/SAS + 4 SATA/SAS/NVMe. Pengguna juga dapat menambahkan RAID card PCIe sendiri di slot PCIe depan yang sudah disediakan.

8 buah drive bay
8 buah drive bay dengan dukungan hingga 4 NVMe

Slot tambahan untuk RAID card
Slot tambahan untuk RAID card


ASUS menyematkan server 2U ini dengan berbagai fitur untuk optimasi performa maupun fitur enterprise.

Performance Tuning

Pengguna dapat menyesuaikan pengaturan optimasi performa menyesuaikan workload secara otomatis berdasarkan profile dari ASUS. Untuk mengakses fitur Performance Tuning ini, pengguna cukup masuk ke System Setup (BIOS) dengan menekan DEL saat startup, lalu masuk ke tab menu Performance Tuning.

Performance Tuning
Performance Tuning

Pengguna dapat memilih setelan optimasi berdasarkan workload maupun benchmark.

Optimasi performa berdasarkan benchmark atau workload tertentu
Optimasi performa berdasarkan benchmark atau workload tertentu

Pada pengujian kali ini EFISON menggunakan opsi By Workload: HPC.

By Workload: HPC
By Workload: HPC

Pengguna juga dapat melakukan overclocking otomatis dengan enable menu overclocking di bawah. Pada pengujian kali ini, EFISON menggunakan setelan Overclocking: Disabled.

BMC WebUI

Pengguna dapat melakukan monitoring kondisi hardware, update firmware, update BIOS, maupun remote control melalui WebUI.

Antarmuka WebUI
Antarmuka WebUI


Pengujian performa dilakukan dengan berbagai software sintetis maupun software saintifik. Berikut adalah konfigurasi hardware dan software yang digunakan.

Tipe Model/spesifikasi
CPU 1 * AMD EPYC 7502P 32c/64t 180W
GPU 1 * Nvidia Tesla T4
RAM 128 GB DDR4-3200 ECC 8-channel
Storage 120 GB Intel SSD DC S3500 Series
OS CentOS 7.8
Kernel 5.9.1-1.el7.elrepo.x86_64

Berikut adalah software yang digunakan dalam pengujian. Seluruh software di-build dari source dengan berbagai optimasi compiler dan library untuk memaksimalkan performa dari software tersebut.

Software Versi Compiler/Library Optimasi
High-Performance LINPACK 2.3 Compiler:
GNU 9.3.0

MPI:
OpenMPI 4.0.4

BLAS:
Intel MKL 2020.0

 Compiler:
-march=znver2

BLAS:
MKL_DEBUG_CPU_TYPE=5
High-Performance Conjugate Gradient 3.1 Compiler:
GNU 9.3.0

MPI:
OpenMPI 4.0.4

(Khusus HPCG GPU, menggunakan binary dengan dukungan CUDA 11 dari web HPCG)

 Compiler:
-march=znver2
GROMACS 2020.3 Compiler:
GNU 9.3.0

MPI:
OpenMPI 4.0.4

BLAS:
BLIS AMD AOCL 2.2 (di-compile sendiri menggunakan GNU 10)

LAPACK:
LibFLAME AMD AOCL 2.2 (di-compile sendiri menggunakan GNU 10)

CUDA:
11.0

 Compiler:
-march=znver2

CUDA:
-arch=sm_75
NAMD 2.14 Compiler:
GNU 9.3.0

FFTW:
Intel MKL 2020.0

CUDA:
11.0

 Compiler:
-march=znver2

BLAS:
MKL_DEBUG_CPU_TYPE=5

CUDA:
-arch=sm_75

High-Performance LINPACK (HPL)

HPL adalah software yang menyelesaikan sistem linier padat acak dalam aritmatik double precision (64-bit) pada komputer dengan memori terdistribusi. HPL merupakan standar benchmark high-performance computing (HPC) dan superkomputer di dunia. Benchmark HPL juga digunakan sebagai tolok ukur performa 500 superkomputer tercepat di dunia yang dirangkum pada laman Top500.

Hasil pengujian HPL
Hasil pengujian HPL

Hasil pengujian HPL menghasilkan skor 1319.3 GFLOPS. Hasil yang dicatatkan oleh EPYC 7502P ini termasuk kencang apabila dibandingkan dengan berbagai CPU kelas workstation maupun enterprise. Sebagai perbandingan, berikut adalah hasil benchmark HPL oleh Dr. Donald Kinghorn dari Puget Systems terhadap berbagai CPU.

Hasil benchmark HPL dari Puget Systems
Hasil benchmark HPL pada berbagai CPU dari Puget Systems

High-Performance Conjugate Gradient (HPCG)

HPCG adalah software benchmark yang melakukan iterasi gradien konjugasi prakondisi multigrid menggunakan nilai floating-point double precision (64-bit). HPCG umum digunakan sebagai suplemen benchmark HPL dan menjadi ukuran efisiensi performa superkomputer dengan perhitungan (hasil HPCG dalam FLOPS/hasil HPL dalam FLOPS).

Pada pengujian HPCG, digunakan konfigurasi problem size 104 104 104 agar HPCG juga bekerja menguji memori dan tidak hanya berjalan di cache prosesor.

Konfigurasi hpcg.dat
Konfigurasi hpcg.dat

HPCG CPU

Hasil benchmark HPCG CPU
Hasil pengujian HPCG CPU

Pengujian HPCG CPU menghasilkan skor dari AMD EPYC 7502P sebesar 14.6655 GFLOPS. Sebagai pembanding, berikut adalah data hasil benchmark HPCG CPU dari Puget Systems.

Hasil benchmark HPCG pada berbagai CPU dari Puget Systems
Hasil benchmark HPCG pada berbagai CPU dari Puget Systems

Skor EPYC 7502P relatif lebih cepat dibanding Threadripper berkat keunggulan 8-channel memori dibanding Threadripper yang hanya 4-channel.

HPCG GPU

 

Hasil pengujian HPCG GPU
Hasil pengujian HPCG GPU

Pengujian HPCG GPU menghasilkan skor dari Nvidia Tesla T4 sebesar 43.4717 GFLOPS. Sebagai pembanding berikut adalah data hasil benchmark HPCG GPU dari Puget Systems.

Hasil benchmark HPCG pada berbagai GPU dari Puget Systems
Hasil benchmark HPCG pada berbagai GPU dari Puget Systems

Skor Nvidia Tesla T4 memang jauh lebih rendah apabila dibandingkan dengan GPU kelas konsumer high-end, namun perlu diingat pula Nvidia Tesla T4 memiliki ukuran yang lebih ringkas (hanya 1 slot PCIe) serta konsumsi daya maksimum yang lebih rendah. Hal ini akan sangat membantu apabila pengguna ingin memasang konfigurasi banyak GPU pada server ASUS ESC4000A-E10 ini.

GROMACS

GROMACS adalah software saintifik untuk melakukan perhitungan dinamika molekuler seperti mensimulasikan persamaan gerak Newton pada sistem dengan jutaan partikel. Ia didesain untuk molekul biokimia seperti protein, lipid, dan asam nukleat yang memiliki banyak interaksi ikatan kompleks.

Pengujian dilakukan menggunakan file input RNAse dodecahedron PME.

GROMACS CPU

GROMACS RNAse dodecahedron PME hanya menggunakan CPU
GROMACS RNAse dodecahedron PME hanya menggunakan CPU

Pada pengujian GROMACS RNAse menggunakan EPYC 7502P, didapatkan hasil hingga 130.709 ns/day. Task hanya menggunakan hingga 32 core untuk mempopulasi jumlah core fisik. Penggunaan hanya core fisik tanpa SMT menghasilkan performa yang lebih cepat dibandingkan dengan mempopulasi seluruh thread.

GROMACS CPU + GPU

GROMACS RNAse menggunakan CPU + GPU
GROMACS RNAse dodecahedron PME menggunakan CPU + GPU

Penggunaan GPU Nvidia Tesla T4 pada pengujian GROMACS RNAse dodecahedron PME menghasilkan akselerasi negatif. Hal ini dikarenakan jumlah GPU yang digunakan terlalu sedikit sehingga off-load tugas dari CPU ke GPU justru lebih tidak efisien. Terlihat bahwa wall time boros di Wait PME GPU gather dan Wait GPU NB local. Hasil pengujian mencatatkan hanya 86.739 ns/day, lebih kecil dibanding hanya menggunakan CPU.

NAMD

NAMD merupakan software saintifik dinamika molekuler paralel yang didesain untuk melakukan simulasi dari sistem biomolekuler besar. Software ini mampu scaling hingga ratusan core untuk simulasi biasa dan lebih dari 500.000 core untuk simulasi besar.

Pengujian dilakukan menggunakan file input ApoA1. Hasil ditunjukkan dalam bentuk skalabilitas terhadap jumlah core.

NAMD CPU

NAMD ApoA1 hanya CPU
NAMD ApoA1 hanya menggunakan CPU

Komputasi NAMD menggunakan AMD EPYC 7502P tanpa dibantu Nvidia Tesla T4 menghasilkan performa yang scaling hingga jumlah task NAMD sama dengan jumlah core fisik (32 core). Pada penggunaan 32 core, 7502P menghasilkan 9.69558 ns/day sedangkan penggunaan maksimum 64 task menghasilkan 10.2963 ns/day.

NAMD CPU + GPU

Hasil NAMD ApoA1 menggunakan CPU + GPU
Hasil NAMD ApoA1 menggunakan CPU + GPU

Komputasi NAMD menggunakan EPYC 7502P dengan dibantu Nvidia Tesla T4 menghasilkan performa komputasi yang jauh lebih baik, hingga lebih dari 32 ns/day.


Server ASUS ESC4000A-E10 ini merupakan server 2U menarik dengan berbagai fitur yang layak untuk digunakan dalam ekosistem HPC. Fitur optimasi performa menggunakan profile workload maupun software di BIOS, dukungan CPU AMD EPYC Rome dengan memori 8-channel DDR4-3200, serta slot PCIe 4.0 hingga 8 GPU menjadikan server ini sangat layak untuk dijadikan pilihan komputasi berat seperti saintifik, machine learning, AI inference, hingga workstation.

Terkait ketersediaan, ASUS menjanjikan server ini hadir di pasar Indonesia antara Q4 2020 atau Q1 2021. Mengenai prosesor AMD EPYC Milan (7003-series) yang akan datang kemungkinan akan didukung, tentunya hal ini menunggu pula informasi rilis dari AMD maupun ASUS.

The post Review ASUS ESC4000A-E10, server 2U dengan AMD EPYC dan Nvidia Tesla untuk HPC appeared first on Efison Lisan Teknologi.

]]> Optimizing GNU Octave 5.2.0 with Intel Math Kernel Library https://efisonlt.com/optimizing-gnu-octave-5-2-0-with-intel-math-kernel-library/?utm_source=rss&utm_medium=rss&utm_campaign=optimizing-gnu-octave-5-2-0-with-intel-math-kernel-library Thu, 22 Oct 2020 05:38:33 +0000 https://efisonlt.com/?p=847 Overview GNU Octave is high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. See GNU Octave Wiki for more info. Building from Source GNU Octave has several… Read More »Optimizing GNU Octave 5.2.0 with Intel Math Kernel Library

The post Optimizing GNU Octave 5.2.0 with Intel Math Kernel Library appeared first on Efison Lisan Teknologi.

]]> Overview

GNU Octave is high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. See GNU Octave Wiki for more info.

Building from Source

GNU Octave has several build dependencies which can be seen here. This blog writing will only cover BLAS, LAPACK, and FFTW3 libraries. Mainly because those three are things which can be accelerated by Intel Math Kernel Library (Intel MKL) while the default option is to stick with free libraries like Netlib’s BLAS/OpenBLAS, Netlib’s LAPACK, and FFTW.

While it is possible to test using those free libraries mentioned as suggested on the build directive from the Wiki, I chose to use AMD AOCL as free library alternative apart from Intel MKL. AMD AOCL was chosen because of its generally better performance compared to any free math libraries counterpart in our internal tests.

Linking GNU Octave with Intel MKL’s FFT

By default, GNU Octave’s configure file doesn’t support linking with Intel MKL’s FFT and would fall back to FFTPACK which is much slower. To be able to utilize MKL version of FFTW3 and FFTW3F, you have to modify the existing configure file manually.

Before editing, generate your own MKL library linking line from Intel Math Kernel Library Link Line Advisor. Make sure to choose correctly every configuration regarding your target machine OS, compiler, architecture, etc.

For example, this is my configuration as I used GNU compiler 9.3.0 and GNU OpenMP:

My Intel MKL linking line
Intel MKL linking line example

Therefore, my linking line woud be:

-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl

After that, put that linking line to configure file. You can find configure in the root dir of the source.

Root dir of GNU Octave source
Root dir of GNU Octave source

Now, update configure. Find these exact line which helps linking Octave with FFTW3 and FFTW3F libraries respectively:

  ac_octave_fftw3_pkg_check=no
  FFTW3_LIBS=
  warn_fftw3="FFTW3 library not found.  The slower FFTPACK library will be used instead."
  case $with_fftw3 in
    no)
      warn_fftw3="--without-fftw3 specified.  Functions or features that depend on FFTW3 will be disabled."
         FFTW3_LIBS=
    ;;
    yes | "")
      ac_octave_fftw3_pkg_check=yes
      FFTW3_LIBS="-lfftw3"
    ;;
    -* | */* | *.a | *.so | *.so.* | *.o)
      FFTW3_LIBS="$with_fftw3"
    ;;
    *)
      FFTW3_LIBS="-l$with_fftw3"
    ;;
  esac

and

  ac_octave_fftw3f_pkg_check=no
  FFTW3F_LIBS=
  warn_fftw3f="FFTW3F library not found.  The slower FFTPACK library will be used instead."
  case $with_fftw3f in
    no)
      warn_fftw3f="--without-fftw3f specified.  Functions or features that depend on FFTW3F will be disabled."
         FFTW3F_LIBS=
    ;;
    yes | "")
      ac_octave_fftw3f_pkg_check=yes
      FFTW3F_LIBS="-lfftw3f"
    ;;
    -* | */* | *.a | *.so | *.so.* | *.o)
      FFTW3F_LIBS="$with_fftw3f"
    ;;
    *)
      FFTW3F_LIBS="-l$with_fftw3f"
    ;;
  esac

Then, edit them to:

  ac_octave_fftw3_pkg_check=no
  FFTW3_LIBS=
  warn_fftw3="FFTW3 library not found.  The slower FFTPACK library will be used instead."
  case $with_fftw3 in
    no)
      warn_fftw3="--without-fftw3 specified.  Functions or features that depend on FFTW3 will be disabled."
         FFTW3_LIBS=
    ;;
    yes | "")
      ac_octave_fftw3_pkg_check=yes
      FFTW3_LIBS="-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"
    ;;
    -* | */* | *.a | *.so | *.so.* | *.o)
      FFTW3_LIBS="$with_fftw3"
    ;;
    *)
      FFTW3_LIBS="-l$with_fftw3"
    ;;
  esac

and

  ac_octave_fftw3f_pkg_check=no
  FFTW3F_LIBS=
  warn_fftw3f="FFTW3F library not found.  The slower FFTPACK library will be used instead."
  case $with_fftw3f in
    no)
      warn_fftw3f="--without-fftw3f specified.  Functions or features that depend on FFTW3F will be disabled."
         FFTW3F_LIBS=
    ;;
    yes | "")
      ac_octave_fftw3f_pkg_check=yes
      FFTW3F_LIBS="-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"
    ;;
    -* | */* | *.a | *.so | *.so.* | *.o)
      FFTW3F_LIBS="$with_fftw3f"
    ;;
    *)
      FFTW3F_LIBS="-l$with_fftw3f"
    ;;
  esac

Next, run autoreconf to reconfigure.

$ autoreconf

Finally, you’ll be able to link Intel MKL’s FFT as GNU Octave FFTW3 and FFTW3F dependencies by simply directing to Intel MKL lib and include directory. Also make sure to link BLAS and LAPACK using linking line generated above.

$ ./configure \
--with-fftw3-includedir=$MKLROOT/include/fftw \
--with-fftw3-libdir=$MKLROOT/lib/intel64 \
--with-fftw3f-includedir=$MKLROOT/include/fftw \
--with-fftw3f-libdir=$MKLROOT/lib/intel64 \
--with-blas="-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl" \
--with-lapack="-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl" \
...

Performance

To find out how good is the speedup from using Intel MKL, I decided to give it a quick test versus FOSS counterpart. Listed below is the machine configuration to test against. For hardware, I used ASUS ESC4000A-E10 provided by ASUS (Thank you, ASUS!).

Type Model
CPU AMD EPYC 7502P
RAM 8-channel 128GB DDR4-3200
OS CentOS 7.7
Kernel 4.4.238-1.el7.elrepo.x86_64

Also, below is compiler and libraries used in the tests.

Type Intel MKL FOSS Notes
Compiler GNU compiler 9.3.0 GNU compiler 9.3.0 FLAGS=-march=znver2
BLAS Intel MKL 2020.0 AMD BLIS AOCL 2.2 MKL was tested using MKL_DEBUG_CPU_TYPE=5. Read here for more info.

AMD BLIS was compiled using GCC 10.1.0 and FLAGS=-march=znver2
LAPACK Intel MKL 2020.0 AMD libflame AOCL 2.2 MKL was tested using MKL_DEBUG_CPU_TYPE=5. Read here for more info.

AMD libflame was compiled using GCC 10.1.0 and FLAGS=-march=znver2
FFTW3/FFTW3F Intel MKL 2020.0 AMD Optimized FFTW AOCL 2.2 MKL was tested using MKL_DEBUG_CPU_TYPE=5. Read here for more info.

AMD Optimized FFTW was compiled using GCC 10.1.0 and FLAGS=-march=znver2

I used benchmarking script for GNU Octave courtesy of Harris Georgiou from University of Piraeus which can be found here. The script consisted of several run-time tests including pseudo-inverse matrix, linear equations system, linear regression, singular value decomposition, fast fourier transform, and bubblesort. Iterations are set on 30 with 2000N vector size which can be set on the script itself. I limited the test itself to 32 thread with SLURM scheduler.

fprintf('Benchmark suite 0.9b (.m) - Harris Georgiou (c) 2018\n\n');

clear all;

Nsz=2000;
Nlp=30;

Results show that GNU Octave built with Intel MKL is generally superior than FOSS counterpart. Here are the results:

Test Intel MKL runtime (second, lower is better) FOSS runtime (second, lower is better) Intel MKL speedup vs FOSS (%)
Pseudo-inverse 0.722554 0.917274 126.95%
Linear equation system 0.36448 0.191732 52.60%
Linear regression 0.224669 1.83595 817.18%
Singular value decomposition 2.53434 3.56501 140.67%
Fast fourier transform 0.0452899 0.0784928 173.31%
Bubblesort 43.3166 43.5649 100.57%

As seen above, GNU Octave linked with Intel MKL only lost once against FOSS in Linear equation system test. Most of the time, MKL won by respectable margin ranged from 100.57% in Bubblesort (which maybe just inside margin of error) to a whopping 817.18% in Linear regression. Granted, maybe Linear regression gains more of its performance from better BLAS/LAPACK implementation but we can also see in Fast fourier transform test that MKL’s FFT also beat AMD Optimized FFTW by a healthy 173.31%. Therefore, it’s safe to say that linking GNU Octave with Intel MKL generally gives a better performance compared to FOSS libraries.

Closing Words

GNU Octave is a good software with good support on FOSS libraries. Some people would be glad to just build their GNU Octave around FOSS BLAS/LAPACK/FFTW libraries. But as we can see above, by adjusting GNU Octave to take advantage of Intel Math Kernel Library (which of course not a FOSS) we can improve its performance up to 8X compared to FOSS. Hopefully this writing can help you build your own Octave with Intel MKL linking and get the most of your machine performance.

The post Optimizing GNU Octave 5.2.0 with Intel Math Kernel Library appeared first on Efison Lisan Teknologi.

]]>