2025, and AI.
What’s not to love?
Again, if somebody were to sell a rendang and they state that it was created using AI, I bet venture capitals would clap and circle like vultures.
Okay enough yapping. Now we are talking about a damn GPU. A tool to run real AI.
Introducing, AMD Radeon AI Pro R9700
In their wisdom, calling a prosumer/workstation GPU with a “pro” moniker doesn’t quite cut it anymore. It now has AI in its name. Why, do you ask?
Because apparently it has 32GB worth of VRAM and it supports another numerical precision (FP8). The VRAM is quite big, we reckon. Still not as big as the last gen Pro (no AI) W7900, but with all its new-ness, AI-ness, and goodness, it’s still pretty nice.
| Radeon AI Pro R9700 | Radeon Pro W7900 | |
|---|---|---|
| Architecture | AMD RDNA 4 | AMD RDNA 3 |
| Memory size | 32 GB | 48 GB |
| Memory bandwidth | 644.6 GB/s | 864 GB/s |
| Memory ECC support | Yes (Linux only)* | Yes |
| Peak FP32 (vector) performance | 47.8 TFLOPS | 61.3 TFLOPS |
| Peak FP16 (vector) performance | 95.7 TFLOPS | 123 TFLOPS |
| Peak FP16 (matrix) performance | 191 TFLOPS | 123 TFLOPS |
| Peak FP8 (matrix) performance | 383 TFLOPS | N/A |
| Peak INT8 (matrix) performance | 383 TOPS | 123 TOPS |
Personally we’re pretty confused about why the memory ECC support on R9700 is stated as Linux only but as we’re mostly using Linux as our test platform, no complaint there.
Even with smaller memory size, it is pretty beefy in terms of performance. Especially in theoretical matrix performance. Hence the AI namesake. Unfortunately we have no W7900 or any RX 7900 XTX for real-world comparison purpose, but we do have an RTX 5070 Ti which apparently has a similar profile.
| Radeon AI Pro R9700 | Geforce RTX 5070 Ti | |
|---|---|---|
| Process technology | TSMC N4P | TSMC 4N |
| Die size | 357 mm² | 378 mm² |
| Memory size | 32 GB GDDR6 | 16 GB GDDR7 |
| Memory interface | 256-bit | 256-bit |
| Memory bandwidth | 644.6 GB/s | 896 GB/s |
| Total board power | 300 W | 300 W |
The Test Setup
| Component | Specification |
|---|---|
| CPU | Intel Core i7-12700K @5.0 GHz |
| Motherboard | ASRock Z690 PG Velocita |
| Memory | 4*24 GB Klevv Cras V RGB @DDR5-5600 |
| Storage | 2TB MSI Spatium M480 |
| PSU | 1000W 1stPlayer NGDP Gold |
For the benchmark, we tested LLM using llama.cpp and image generation using Qwen Image Edit 2509.
llama.cpp Benchmark Setup
We used llama.cpp build d2ee056e1 (6713) and compiled the CPU backend using Intel oneAPI compiler 2025.2.1 against external BLAS library which is Intel oneAPI MKL 2025.2. Why, do you ask? Because it yields faster performance compared to mere GNU compiler 15.2.1 with no BLAS.
We tested using unsloth/gpt-oss-120b-Q4_K_M model and .(7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(up|down|gate)_exps. MoE layers which are then being put to system RAM for CPU offload processing.
| (In token/s. Higher is better) | GNU compiler 15.2.1 no BLAS | oneAPI compiler 2025.2.1 BLAS=oneAPI MKL 2025.2 |
|---|---|---|
| Prompt processing (512 tokens) | 180.65 ± 1.74 | 182.39 ± 1.60 |
| Text generation (256 tokens) | 21.85 ± 0.82 | 32.19 ± 0.04 |
Compilation steps for GNU compiler 15.2.1, no BLAS
mkdir build_no-blas-gcc_vulkan && cd build_no-blas-gcc_vulkan
cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -jCompilation steps for oneAPI compiler 2025.2.1, BLAS=oneAPI MKL 2025.2
mkdir build_mkl-ilp64-icx_vulkan && build_mkl-ilp64-icx_vulkan
cmake -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -jRun output
GGML_VULKAN_DEVICE=0 ./build_no-blas-gcc_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | ngl | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan | 99 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 180.65 ± 1.74 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan | 99 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 21.85 ± 0.82 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 182.39 ± 1.60 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 32.19 ± 0.04 |
build: d2ee056e1 (6713)For the GPU backend, we used both Vulkan and HIP (ROCm 7) which we won’t discuss much here to prevent spoilers.
We refrained on using ROCm 6.4.x as ROCm 7.0.x is now performing much better on this GPU (or probably all AMD RDNA4 GPUs in general). If you haven’t heard already, AMD have just released their newest ROCm 7 on September 16, 2025. We did a quick comparison in terms of llama.cpp performance against ROCm 6.4.3 which can be seen on this guy’s Facebook post.
As for the RTX 5070 Ti, we compiled llama.cpp GPU backend against CUDA without GGML_CUDA_FORCE_CUBLAS.
So for references, here are the compilation lines used for various configurations stated above:
Compilation line for llama.cpp ROCm
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_HIP=ON -DGPU_TARGETS=gfx1201Compilation line for llama.cpp Vulkan
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1Compilation line for llama.cpp CUDA
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_CUDA=ONWe chose 2 models to be used for testing:
- unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS, 15.25 GB
- unsloth/gpt-oss-120b-Q4_K_M, 58.45 GB
Qwen Image Edit 2509 Benchmark Setup (and rambling about PyTorch for ROCm on Windows situation)
This one is a bit different because it has PyTorch dependencies and it’s not that simple. Historically, AMD has been neglecting (or unable to make?) Windows PyTorch package for their GPUs. With the Windows Preview Edition 25.20.01.14 driver, they finally support PyTorch on Windows for Radeons. Yay!
Albeit the limited roster of supported GPUs…
Funnily enough, they only listed Windows 11 as the compatible OS. Lo and behold, we managed to use it on Windows 10.
Enough with the rambling, now let’s get onto the setup.
We did the tests on 2 different OSes:
- Aurora Linux 42 based on Fedora Kinoite
- Windows 10 Pro
For the PyTorch, we tested different combinations:
| PyTorch 2.x for ROCm 6.4.x | PyTorch 2.x for ROCm 7.0.x | |
|---|---|---|
| Aurora Linux 42 | ✘ | ✔ |
| Windows 10 Pro | ✔ | ✔ |
The Qwen Image Edit 2509 models used here were QuantStack/Qwen-Image-Edit-2509-Q4_K_M. We also used lightx2v/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16 LoRA for performance reason. As for the UI and additional package, we used ComfyUI v0.3.64. Not to forgot to mention city96/ComfyUI-GGUF for GGUF models compatibility.
Additional notes about Qwen Image Edit 2509 test setup and deployment
- We deployed the ComfyUI with ROCm support on Linux using Podman container based on Ubuntu 22.04 ROCm 7.0 Docker image, in which the compose scripts can be cloned from git lslowmotion/stable-diffusion-webui-podman
- ComfyUI with ROCm support on Windows was deployed using ComfyUI experimental portable package for AMD GPUs which has PyTorch 2.x for ROCm 6.4.x included, and ROCm 7.0.x tests were done by manually upgrade the PyTorch PIP package to nightly package targeting gfx120x which is the arch code for the Radeon AI Pro R9700.
As for the inputs, we used these images:
And these sentences for the prompt:
Test Results: llama.cpp
We talked a bit about llama.cpp LLM MoE layer offload to CPU above. But in short, CPU offloading is done to make sure you can run a big model without the big performance hit caused by memory spill from VRAM to system RAM.
So, if you run a big model on GPU, and the VRAM is smaller than the model, it will still run. But it will run terribly because now the GPU needs to access the data from RAM in which the available VRAM can’t contain, while the CPU is doing nothing to help the processing.
Now see the difference with CPU offloading.
Probably this article is not the best at explaining how the MoE layers work or the way they’re offloaded. You can read a little bit more technical stuffs starting from reading this guide, or maybe a little bit of Google-fu. Sorry.
Now, let’s see the performance difference:
| (In token/s. Higher is better) | Without CPU offloading | With CPU offloading |
|---|---|---|
| Prompt processing (512 tokens) | 120.48 ± 4.06 | 215.93 ± 1.42 |
| Text generation (256 tokens) | 11.21 ± 0.02 | 34.97 ± 0.15 |
Run output
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | pp512 | 120.48 ± 4.06 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | tg256 | 11.21 ± 0.02 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | pp512 | 215.93 ± 1.42 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | tg256 | 34.97 ± 0.15 |
build: d2ee056e1 (6713)Let’s continue with the test results.
Here we have different configs of layers for CPU offloading. One is to keep the VRAM usage to under 16 GB, and then maximize the possible 32 GB. The configuration details can be seen below:
- RTX 5070 Ti with just under 16 GB VRAM load, CUDA backend
- RTX 5070 Ti with just under 16 GB VRAM load, Vulkan backend
- R9700 with just under 16 GB VRAM load, ROCm 7 backend
- R9700 with just under 16 GB VRAM load, Vulkan backend
- R9700 with just under 32 GB VRAM load, ROCm 7 backend
- R9700 with just under 32 GB VRAM load, Vulkan backend
Here are the results using unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS:
| (In token/s. Higher is better) | 5070 Ti, 16G CUDA | 5070 Ti, 16G Vulkan | R9700, 16G ROCm 7 | R9700, 16G Vulkan | R9700, 32G ROCm 7 | R9700, 32G Vulkan |
|---|---|---|---|---|---|---|
| Prompt processing (512 tokens) | 3723.33 ± 50.09 | 2739.83 ± 30.37 | 746.77 ± 3.45 | 1236.97 ± 9.19 | 797.92 ± 3.61 | 1665.47 ± 5.95 |
| Text generation (256 tokens) | 137.34 ± 0.69 | 138.41 ± 0.79 | 88.98 ± 0.16 | 105.35 ± 0.75 | 100.05 ± 0.08 | 122.63 ± 0.52 |
Here are the results using unsloth/gpt-oss-120b-Q4_K_M:
| (In token/s. Higher is better) | 5070 Ti, 16G CUDA | 5070 Ti, 16G Vulkan | R9700, 16G ROCm 7 | R9700, 16G Vulkan | R9700, 32G ROCm 7 | R9700, 32G Vulkan |
|---|---|---|---|---|---|---|
| Prompt processing (512 tokens) | 370.32 ± 4.22 | 206.81 ± 3.52 | 188.32 ± 4.82 | 169.56 ± 2.60 | 251.93 ± 6.61 | 230.01 ± 2.78 |
| Text generation (256 tokens) | 40.24 ± 0.13 | 37.90 ± 0.42 | 32.59 ± 0.01 | 31.49 ± 0.04 | 38.73 ± 0.08 | 36.22 ± 0.03 |
RTX 5070 Ti 16G run output
./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | CUDA,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | pp512 | 3723.33 ± 50.09 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | CUDA,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | tg256 | 137.34 ± 0.69 |
build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | pp512 | 2739.83 ± 30.37 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | tg256 | 138.41 ± 0.79 |
build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | CUDA,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 370.32 ± 4.22 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | CUDA,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 40.24 ± 0.13 |
build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 206.81 ± 3.52 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 37.90 ± 0.42 |
build: d2ee056e1 (6713)
R9700 16G run output
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | pp512 | 746.77 ± 3.45 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | tg256 | 88.98 ± 0.16 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | pp512 | 1236.97 ± 9.19 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU | tg256 | 105.35 ± 0.75 |
build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 188.32 ± 4.82 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 32.59 ± 0.01 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | pp512 | 169.56 ± 2.60 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU | tg256 | 31.49 ± 0.04 |
build: d2ee056e1 (6713)
R9700 32G run output
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | pp512 | 797.92 ± 3.61 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | tg256 | 100.05 ± 0.08 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | pp512 | 1665.47 ± 5.95 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | tg256 | 122.63 ± 0.52 |
build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | pp512 | 251.93 ± 6.61 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | tg256 | 38.73 ± 0.08 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | pp512 | 230.01 ± 2.78 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | tg256 | 36.22 ± 0.03 |
build: d2ee056e1 (6713)
Even though it has twice the VRAM, with the MoE layers CPU offloading strategy, turns out it still can’t quite compete with the RTX 5070 Ti. This might be caused by the large memory bandwidth gap between both (644.6 GB/s vs 896 GB/s, 39% difference).
Now, when you see on unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS, it has this big performance delta between ROCm 7 backend and Vulkan backend. I wonder why a bunch of open-source developers from Vulkan/SPIR-V/Kompute project can outperform engineers who are paid to make their stuffs performant… Although, it did better on unsloth/gpt-oss-120b-Q4_K_M by a tiny margin.
Also, turns out having a bigger VRAM still can’t quite defeat the smaller VRAM, if your bandwidth is too slow, and your software isn’t up to par.
It’s not about the size. It’s about how you use it.
Test Results: Qwen Image Edit 2509
Sometimes we question ourselves why do we like to suffer.
And this test is no different.
We (or me, personally), wonder if being a goose farmer is a better choice for living a happy life.
First, let us tell you that running this test for the first run, is not a happy feat. It takes quite a while for the text encoder and VAE to load and process our prompt and image inputs.
UPDATE: IT DOESN’T HAVE TO BE 7 MINUTES
PyTorch for ROCm apparently has a bug in which the VAE stage is extremely slow.
The solution? ComfyUI recently pushed a workaround. Which is to disable cuDNN back-end (?).
Now the first generation isn’t that painful anymore.
Original article continues below.
Here, we only put the generation time after text encoder and VAE had been loaded and then run it in a batch of 5. Then we averaged the time needed to generate the same prompt with the same inputs.
Now, let’s get onto the results:
| (In seconds, Lower is better) | RTX 5070 Ti CUDA 12.9 Linux | R9700 ROCm 7.0.x Linux | R9700 ROCm 6.4.x Windows | R9700 ROCm 7.0.x Windows |
|---|---|---|---|---|
| Results | 29.384 | 52.17 | 69.262 | 62.59 |
RTX 5070 Ti CUDA 12.9 Linux test screenshot
https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-GGUF.png
R9700 ROCm 7.0.x Linux test screenshot
https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF.png
R9700 ROCm 6.4.x Windows test screenshot
https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF.png
R9700 ROCm 7.0.x Windows test screenshot
https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF-gfx120x-nightly.png
Oops.
Almost twice faster.
Also you can make a case that PyTorch for ROCm on Windows is slower. Almost 20% slower than on Linux. Although this is only on one use case that we tested and we didn’t confirm with different various use cases.
And to make it worse, RTX 5070 Ti does have a trick up its sleeve.
Introducing Nunchaku SVDQuant. An inference engine so fast it cut the inference time of Qwen Image Edit 2509 almost in half even versus already quantized Q4_K_M model.
| (In seconds, Lower is better) | RTX 5070 Ti CUDA 12.9 Linux Nunchaku FP4 r32 | RTX 5070 Ti CUDA 12.9 Linux Q4_K_M GGUF |
|---|---|---|
| Results | 15.198 | 29.384 |
RTX 5070 Ti CUDA 12.9 Linux, Nunchaku FP4 r32 model test screenshot
https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-Nunchaku-r32-CPU-offload.png
Overclocking
Real men do OC.
Or men with too much times in hands.
We managed to overclock this card using LACT. No, we didn’t test overclocking on Windows. Penguins FTW!
We found out that 2800 MHz was the ideal maximum VRAM clock to keep it from crashing doing various workloads. Still a healthy 282 MHz increase from the default 2518 MHz. Which translates to 11.2% more memory bandwidth.
The maximum power usage limit also increased to 330 W from 300 W (10% increase). And maybe you wonder what does GPU voltage offset about? Why is it being lowered, right?
Some of you might have heard that overclocking the GPU clock on RDNA4 is done by shifting the voltage/frequency curve by setting the voltage offset to a lower value, so the GPU would be tricked and boosts to higher frequency.
Too difficult to understand? Let us show you a video from our friend Alva Jonathan who did an excellent job explaining overclocking on another RDNA4 GPU which is a Radeon RX 9070.
Here are the performance results to show you the gain from the overclocking attempt:
llama.cpp unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS (Overclocked)
| (In token/s. Higher is better) | 5070 Ti, 16G CUDA | R9700, 32G Vulkan Overclocked | R9700, 32G Vulkan Stock default |
|---|---|---|---|
| Prompt processing (512 tokens) | 3723.33 ± 50.09 | 1810.00 ± 11.36 | 1665.47 ± 5.95 |
| Text generation (256 tokens) | 137.34 ± 0.69 | 131.09 ± 0.37 | 122.63 ± 0.52 |
llama.cpp unsloth/gpt-oss-120b-Q4_K_M (Overclocked)
| (In token/s. Higher is better) | 5070 Ti, 16G CUDA | R9700, 32G ROCm 7 Overclocked | R9700, 32G ROCm 7 Stock default |
|---|---|---|---|
| Prompt processing (512 tokens) | 370.32 ± 4.22 | 254.46 ± 4.62 | 251.93 ± 6.61 |
| Text generation (256 tokens) | 40.24 ± 0.13 | 39.72 ± 0.04 | 38.73 ± 0.08 |
Overclocked R9700 llama.cpp run output
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | pp512 | 834.27 ± 3.99 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | tg256 | 105.42 ± 0.12 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | pp512 | 1810.00 ± 11.36 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw | 15.25 GiB | 30.53 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | tg256 | 131.09 ± 0.37 |
build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | pp512 | 254.46 ± 4.62 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | ROCm,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | tg256 | 39.72 ± 0.04 |
build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model | size | params | backend | threads | n_batch | n_ubatch | type_k | type_v | fa | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | pp512 | 236.58 ± 2.89 |
| gpt-oss 120B Q4_K - Medium | 58.45 GiB | 116.83 B | Vulkan,BLAS | 8 | 4096 | 4096 | q8_0 | q8_0 | 1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU | tg256 | 37.41 ± 0.04 |
build: d2ee056e1 (6713)As we can see, the increase is bigger when the model is smaller, as bigger portion of layers fit on the VRAM hence the performance gain from VRAM overclocking. Still a tad slower than RTX 5070 Ti, but we’ll grab what we can.
Qwen Image Edit 2509 Q4_K_M GGUF (Overclocked)
| (In seconds, Lower is better) | RTX 5070 Ti CUDA 12.9 Linux | R9700 ROCm 7.0.x Linux Overclocked | R9700 ROCm 7.0.x Linux Stock default |
|---|---|---|---|
| Results | 29.384 | 48.628 | 52.17 |
Overclocked R9700 run screenshot
https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF-OC.png
7.28% faster. Not bad for a free performance gain.
Verdict
So, what do we think?
Bigger VRAM doesn’t always translate to a bigger performance. Especially in our tests which are very inference heavy. This doesn’t mean this card is DoA or something (pls don’t be, we need alternatives to the leather jacketed overlord).
There are many good takeaways. Such as AMD GPU software team are now finally getting their stuffs together. PyTorch for ROCm on Windows was one of the biggest to-do list. ROCm 7 is being faster than ROCm 6 (at least on RDNA4).
They also managed to lower the size of ROCm Docker image and PyTorch package.
Unfortunately, turns out it was due to they chose to drop RDNA2 from their supported GPUs. Sad.
We still think there would be a better value for a bigger VRAM for different use cases. Probably in LLM finetuning, in 3D modelling/rendering, which we presume wouldn’t be easier to work around with CPU offload the way we tested MoE LLM inference above.
