comfyui Archives - Efison Lisan Teknologi https://efisonlt.com/tag/comfyui/ Computation for Everybody Thu, 28 May 2026 10:13:53 +0000 en-US hourly 1 https://wordpress.org/?v=7.0 https://efisonlt.com/wp-content/uploads/2020/03/cropped-efison_logo_orange-skuer-32x32.png comfyui Archives - Efison Lisan Teknologi https://efisonlt.com/tag/comfyui/ 32 32 Our Experience with Asus AMD Radeon AI Pro R9700 Turbo https://efisonlt.com/our-experience-with-asus-amd-radeon-ai-pro-r9700-turbo/?utm_source=rss&utm_medium=rss&utm_campaign=our-experience-with-asus-amd-radeon-ai-pro-r9700-turbo Sat, 11 Oct 2025 13:13:19 +0000 https://efisonlt.com/?p=1880 2025, and AI. What’s not to love? Again, if somebody were to sell a rendang and they state that it was created using AI, I bet venture capitals would clap and circle like vultures. Okay enough yapping. Now we are talking about a damn GPU. A tool to run real AI. Introducing, AMD Radeon AI… Read More »Our Experience with Asus AMD Radeon AI Pro R9700 Turbo

The post Our Experience with Asus AMD Radeon AI Pro R9700 Turbo appeared first on Efison Lisan Teknologi.

]]> 2025, and AI.

What’s not to love?

Again, if somebody were to sell a rendang and they state that it was created using AI, I bet venture capitals would clap and circle like vultures.

Okay enough yapping. Now we are talking about a damn GPU. A tool to run real AI.

Introducing, AMD Radeon AI Pro R9700

In their wisdom, calling a prosumer/workstation GPU with a “pro” moniker doesn’t quite cut it anymore. It now has AI in its name. Why, do you ask?

Because apparently it has 32GB worth of VRAM and it supports another numerical precision (FP8). The VRAM is quite big, we reckon. Still not as big as the last gen Pro (no AI) W7900, but with all its new-ness, AI-ness, and goodness, it’s still pretty nice.

Radeon AI Pro R9700Radeon Pro W7900
ArchitectureAMD RDNA 4AMD RDNA 3
Memory size32 GB48 GB
Memory bandwidth644.6 GB/s864 GB/s
Memory ECC supportYes (Linux only)*Yes
Peak FP32 (vector) performance47.8 TFLOPS61.3 TFLOPS
Peak FP16 (vector) performance95.7 TFLOPS123 TFLOPS
Peak FP16 (matrix) performance191 TFLOPS123 TFLOPS
Peak FP8 (matrix) performance383 TFLOPSN/A
Peak INT8 (matrix) performance383 TOPS123 TOPS

Personally we’re pretty confused about why the memory ECC support on R9700 is stated as Linux only but as we’re mostly using Linux as our test platform, no complaint there.

Even with smaller memory size, it is pretty beefy in terms of performance. Especially in theoretical matrix performance. Hence the AI namesake. Unfortunately we have no W7900 or any RX 7900 XTX for real-world comparison purpose, but we do have an RTX 5070 Ti which apparently has a similar profile.

Radeon AI Pro R9700Geforce RTX 5070 Ti
Process technologyTSMC N4PTSMC 4N
Die size357 mm²378 mm²
Memory size32 GB GDDR616 GB GDDR7
Memory interface256-bit256-bit
Memory bandwidth644.6 GB/s896 GB/s
Total board power300 W300 W

The Test Setup

ComponentSpecification
CPUIntel Core i7-12700K
@5.0 GHz
MotherboardASRock Z690 PG Velocita
Memory4*24 GB Klevv Cras V RGB
@DDR5-5600
Storage2TB MSI Spatium M480
PSU1000W 1stPlayer NGDP Gold

For the benchmark, we tested LLM using llama.cpp and image generation using Qwen Image Edit 2509.

llama.cpp Benchmark Setup

We used llama.cpp build d2ee056e1 (6713) and compiled the CPU backend using Intel oneAPI compiler 2025.2.1 against external BLAS library which is Intel oneAPI MKL 2025.2. Why, do you ask? Because it yields faster performance compared to mere GNU compiler 15.2.1 with no BLAS.

We tested using unsloth/gpt-oss-120b-Q4_K_M model and .(7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(up|down|gate)_exps. MoE layers which are then being put to system RAM for CPU offload processing.

(In token/s. Higher is better)GNU compiler 15.2.1
no BLAS		oneAPI compiler 2025.2.1
BLAS=oneAPI MKL 2025.2
Prompt processing (512 tokens)180.65 ± 1.74182.39 ± 1.60
Text generation (256 tokens)21.85 ± 0.8232.19 ± 0.04
Compilation steps for GNU compiler 15.2.1, no BLAS
Bash
mkdir build_no-blas-gcc_vulkan && cd build_no-blas-gcc_vulkan
cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -j
Compilation steps for oneAPI compiler 2025.2.1, BLAS=oneAPI MKL 2025.2
Bash
mkdir build_mkl-ilp64-icx_vulkan && build_mkl-ilp64-icx_vulkan
cmake -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1
cmake --build . --config Release -j
Run output
Bash
GGML_VULKAN_DEVICE=0 ./build_no-blas-gcc_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan     |  99 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        180.65 ± 1.74 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan     |  99 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         21.85 ± 0.82 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        182.39 ± 1.60 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         32.19 ± 0.04 |

build: d2ee056e1 (6713)

For the GPU backend, we used both Vulkan and HIP (ROCm 7) which we won’t discuss much here to prevent spoilers.

We refrained on using ROCm 6.4.x as ROCm 7.0.x is now performing much better on this GPU (or probably all AMD RDNA4 GPUs in general). If you haven’t heard already, AMD have just released their newest ROCm 7 on September 16, 2025. We did a quick comparison in terms of llama.cpp performance against ROCm 6.4.3 which can be seen on this guy’s Facebook post.

As for the RTX 5070 Ti, we compiled llama.cpp GPU backend against CUDA without GGML_CUDA_FORCE_CUBLAS.

So for references, here are the compilation lines used for various configurations stated above:

Compilation line for llama.cpp ROCm
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_HIP=ON -DGPU_TARGETS=gfx1201
Compilation line for llama.cpp Vulkan
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_VULKAN=1
Compilation line for llama.cpp CUDA
Bash
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64ilp -DGGML_NATIVE=ON -DGGML_CUDA=ON

We chose 2 models to be used for testing:

Qwen Image Edit 2509 Benchmark Setup (and rambling about PyTorch for ROCm on Windows situation)

This one is a bit different because it has PyTorch dependencies and it’s not that simple. Historically, AMD has been neglecting (or unable to make?) Windows PyTorch package for their GPUs. With the Windows Preview Edition 25.20.01.14 driver, they finally support PyTorch on Windows for Radeons. Yay!

Albeit the limited roster of supported GPUs…

Funnily enough, they only listed Windows 11 as the compatible OS. Lo and behold, we managed to use it on Windows 10.

Enough with the rambling, now let’s get onto the setup.

We did the tests on 2 different OSes:

  • Aurora Linux 42 based on Fedora Kinoite
  • Windows 10 Pro

For the PyTorch, we tested different combinations:

PyTorch 2.x for ROCm 6.4.xPyTorch 2.x for ROCm 7.0.x
Aurora Linux 42✘✔
Windows 10 Pro✔✔

The Qwen Image Edit 2509 models used here were QuantStack/Qwen-Image-Edit-2509-Q4_K_M. We also used lightx2v/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16 LoRA for performance reason. As for the UI and additional package, we used ComfyUI v0.3.64. Not to forgot to mention city96/ComfyUI-GGUF for GGUF models compatibility.

Additional notes about Qwen Image Edit 2509 test setup and deployment

As for the inputs, we used these images:

And these sentences for the prompt:

Test Results: llama.cpp

We talked a bit about llama.cpp LLM MoE layer offload to CPU above. But in short, CPU offloading is done to make sure you can run a big model without the big performance hit caused by memory spill from VRAM to system RAM.

So, if you run a big model on GPU, and the VRAM is smaller than the model, it will still run. But it will run terribly because now the GPU needs to access the data from RAM in which the available VRAM can’t contain, while the CPU is doing nothing to help the processing.

unsloth/gpt-oss-120b-Q4_K_M has the size of 58.45 GB, while the available VRAM is only around 32 GB before substracting other necessary services for the operating system, which is now spilling around 26-27 GB to system RAM.

Now see the difference with CPU offloading.

With .(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9]).ffn_(up|down)_exps. layers being offloaded to CPU and put to the system RAM, now the CPU can help with the processing while preventing the layers loaded to VRAM to spill over.

Probably this article is not the best at explaining how the MoE layers work or the way they’re offloaded. You can read a little bit more technical stuffs starting from reading this guide, or maybe a little bit of Google-fu. Sorry.

Now, let’s see the performance difference:

(In token/s. Higher is better)Without CPU offloadingWith CPU offloading
Prompt processing (512 tokens)120.48 ± 4.06215.93 ± 1.42
Text generation (256 tokens)11.21 ± 0.0234.97 ± 0.15
Run output
Bash
GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        120.48 ± 4.06 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |         11.21 ± 0.02 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        215.93 ± 1.42 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(4|5|6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         34.97 ± 0.15 |

build: d2ee056e1 (6713)

Let’s continue with the test results.

Here we have different configs of layers for CPU offloading. One is to keep the VRAM usage to under 16 GB, and then maximize the possible 32 GB. The configuration details can be seen below:

  • RTX 5070 Ti with just under 16 GB VRAM load, CUDA backend
  • RTX 5070 Ti with just under 16 GB VRAM load, Vulkan backend
  • R9700 with just under 16 GB VRAM load, ROCm 7 backend
  • R9700 with just under 16 GB VRAM load, Vulkan backend
  • R9700 with just under 32 GB VRAM load, ROCm 7 backend
  • R9700 with just under 32 GB VRAM load, Vulkan backend

Here are the results using unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS:

(In token/s. Higher is better)5070 Ti, 16G
CUDA		5070 Ti, 16G
Vulkan		R9700, 16G
ROCm 7		R9700, 16G
Vulkan		R9700, 32G
ROCm 7		R9700, 32G
Vulkan
Prompt processing (512 tokens)3723.33 ± 50.092739.83 ± 30.37746.77 ± 3.451236.97 ± 9.19797.92 ± 3.611665.47 ± 5.95
Text generation (256 tokens)137.34 ± 0.69138.41 ± 0.7988.98 ± 0.16105.35 ± 0.75100.05 ± 0.08122.63 ± 0.52

Here are the results using unsloth/gpt-oss-120b-Q4_K_M:

(In token/s. Higher is better)5070 Ti, 16G
CUDA		5070 Ti, 16G
Vulkan		R9700, 16G
ROCm 7		R9700, 16G
Vulkan		R9700, 32G
ROCm 7		R9700, 32G
Vulkan
Prompt processing (512 tokens)370.32 ± 4.22206.81 ± 3.52188.32 ± 4.82169.56 ± 2.60251.93 ± 6.61230.01 ± 2.78
Text generation (256 tokens)40.24 ± 0.1337.90 ± 0.4232.59 ± 0.0131.49 ± 0.0438.73 ± 0.0836.22 ± 0.03
RTX 5070 Ti 16G run output
Bash
./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_cuda/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |      3723.33 ± 50.09 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        137.34 ± 0.69 |

build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |      2739.83 ± 30.37 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        138.41 ± 0.79 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        370.32 ± 4.22 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | CUDA,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         40.24 ± 0.13 |

build: d2ee056e1 (6713)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        206.81 ± 3.52 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         37.90 ± 0.42 |

build: d2ee056e1 (6713)
R9700 16G run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |        746.77 ± 3.45 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |         88.98 ± 0.16 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           pp512 |       1236.97 ± 9.19 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(39|[4-9][0-9]|[1-9][0-9][0-9])\.ffn_(gate)_exps.=CPU |           tg256 |        105.35 ± 0.75 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        188.32 ± 4.82 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         32.59 ± 0.01 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           pp512 |        169.56 ± 2.60 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down|gate)_exps.=CPU |           tg256 |         31.49 ± 0.04 |

build: d2ee056e1 (6713)
R9700 32G run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        797.92 ± 3.61 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        100.05 ± 0.08 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |       1665.47 ± 5.95 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        122.63 ± 0.52 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        251.93 ± 6.61 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         38.73 ± 0.08 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        230.01 ± 2.78 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         36.22 ± 0.03 |

build: d2ee056e1 (6713)

Even though it has twice the VRAM, with the MoE layers CPU offloading strategy, turns out it still can’t quite compete with the RTX 5070 Ti. This might be caused by the large memory bandwidth gap between both (644.6 GB/s vs 896 GB/s, 39% difference).

Now, when you see on unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS, it has this big performance delta between ROCm 7 backend and Vulkan backend. I wonder why a bunch of open-source developers from Vulkan/SPIR-V/Kompute project can outperform engineers who are paid to make their stuffs performant… Although, it did better on unsloth/gpt-oss-120b-Q4_K_M by a tiny margin.

Also, turns out having a bigger VRAM still can’t quite defeat the smaller VRAM, if your bandwidth is too slow, and your software isn’t up to par.

It’s not about the size. It’s about how you use it.

Test Results: Qwen Image Edit 2509

Sometimes we question ourselves why do we like to suffer.

And this test is no different.

We (or me, personally), wonder if being a goose farmer is a better choice for living a happy life.

First, let us tell you that running this test for the first run, is not a happy feat. It takes quite a while for the text encoder and VAE to load and process our prompt and image inputs.

On a bad day, it can take more than 7 mins of time to generate this for the first run.

UPDATE: IT DOESN’T HAVE TO BE 7 MINUTES

PyTorch for ROCm apparently has a bug in which the VAE stage is extremely slow.

The solution? ComfyUI recently pushed a workaround. Which is to disable cuDNN back-end (?).

Now the first generation isn’t that painful anymore.

Original article continues below.

Here, we only put the generation time after text encoder and VAE had been loaded and then run it in a batch of 5. Then we averaged the time needed to generate the same prompt with the same inputs.

Now, let’s get onto the results:

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux		R9700
ROCm 7.0.x
Linux		R9700
ROCm 6.4.x
Windows		R9700
ROCm 7.0.x
Windows
Results29.38452.1769.26262.59
RTX 5070 Ti CUDA 12.9 Linux test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-GGUF.png

R9700 ROCm 7.0.x Linux test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF.png

R9700 ROCm 6.4.x Windows test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF.png

R9700 ROCm 7.0.x Windows test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/Windows-R9700-GGUF-gfx120x-nightly.png

Oops.

Almost twice faster.

Also you can make a case that PyTorch for ROCm on Windows is slower. Almost 20% slower than on Linux. Although this is only on one use case that we tested and we didn’t confirm with different various use cases.

And to make it worse, RTX 5070 Ti does have a trick up its sleeve.

Introducing Nunchaku SVDQuant. An inference engine so fast it cut the inference time of Qwen Image Edit 2509 almost in half even versus already quantized Q4_K_M model.

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux
Nunchaku FP4 r32		RTX 5070 Ti
CUDA 12.9
Linux
Q4_K_M GGUF
Results15.19829.384
RTX 5070 Ti CUDA 12.9 Linux, Nunchaku FP4 r32 model test screenshot

https://efisonlt.com/wp-content/uploads/2025/10/5070-Ti-Nunchaku-r32-CPU-offload.png

Overclocking

Real men do OC.

Or men with too much times in hands.

We managed to overclock this card using LACT. No, we didn’t test overclocking on Windows. Penguins FTW!

We found out that 2800 MHz was the ideal maximum VRAM clock to keep it from crashing doing various workloads. Still a healthy 282 MHz increase from the default 2518 MHz. Which translates to 11.2% more memory bandwidth.

The maximum power usage limit also increased to 330 W from 300 W (10% increase). And maybe you wonder what does GPU voltage offset about? Why is it being lowered, right?

Some of you might have heard that overclocking the GPU clock on RDNA4 is done by shifting the voltage/frequency curve by setting the voltage offset to a lower value, so the GPU would be tricked and boosts to higher frequency.

Too difficult to understand? Let us show you a video from our friend Alva Jonathan who did an excellent job explaining overclocking on another RDNA4 GPU which is a Radeon RX 9070.

Here are the performance results to show you the gain from the overclocking attempt:

llama.cpp unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS (Overclocked)

(In token/s. Higher is better)5070 Ti, 16G
CUDA		R9700, 32G
Vulkan
Overclocked		R9700, 32G
Vulkan
Stock default
Prompt processing (512 tokens)3723.33 ± 50.091810.00 ± 11.361665.47 ± 5.95
Text generation (256 tokens)137.34 ± 0.69131.09 ± 0.37122.63 ± 0.52

llama.cpp unsloth/gpt-oss-120b-Q4_K_M (Overclocked)

(In token/s. Higher is better)5070 Ti, 16G
CUDA		R9700, 32G
ROCm 7
Overclocked		R9700, 32G
ROCm 7
Stock default
Prompt processing (512 tokens)370.32 ± 4.22254.46 ± 4.62251.93 ± 6.61
Text generation (256 tokens)40.24 ± 0.1339.72 ± 0.0438.73 ± 0.08
Overclocked R9700 llama.cpp run output
Bash
./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-IQ4_XS.gguf -ctk q8_0 -ctv q8_0  --threads 8 -ngl 99 -p 512 -n 256 -fa 1 -ub 4096 -b 4096

./build_mkl-ilp64-icx_rocm/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096

GGML_VULKAN_DEVICE=0 ./build_mkl-ilp64-icx_vulkan/bin/llama-bench --model ../MoE/unsloth/gpt-oss-120b-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --threads 8 -ngl 99 -ot "\.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU" -p 512 -n 256 -fa 1 -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |        834.27 ± 3.99 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        105.42 ± 0.12 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           pp512 |      1810.00 ± 11.36 |
| qwen3moe 30B.A3B IQ4_XS - 4.25 bpw |  15.25 GiB |    30.53 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 |           tg256 |        131.09 ± 0.37 |

build: d2ee056e1 (6713)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        254.46 ± 4.62 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | ROCm,BLAS  |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         39.72 ± 0.04 |

build: d2ee056e1 (6713)
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (ADL-S GT1) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | threads | n_batch | n_ubatch | type_k | type_v | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -----: | -----: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           pp512 |        236.58 ± 2.89 |
| gpt-oss 120B Q4_K - Medium     |  58.45 GiB |   116.83 B | Vulkan,BLAS |       8 |    4096 |     4096 |   q8_0 |   q8_0 |  1 | \.(7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(up|down)_exps.=CPU |           tg256 |         37.41 ± 0.04 |

build: d2ee056e1 (6713)

As we can see, the increase is bigger when the model is smaller, as bigger portion of layers fit on the VRAM hence the performance gain from VRAM overclocking. Still a tad slower than RTX 5070 Ti, but we’ll grab what we can.

Qwen Image Edit 2509 Q4_K_M GGUF (Overclocked)

(In seconds, Lower is better)RTX 5070 Ti
CUDA 12.9
Linux		R9700
ROCm 7.0.x
Linux
Overclocked		R9700
ROCm 7.0.x
Linux
Stock default
Results29.38448.62852.17
Overclocked R9700 run screenshot

https://efisonlt.com/wp-content/uploads/2025/10/R9700-GGUF-OC.png

7.28% faster. Not bad for a free performance gain.

Verdict

So, what do we think?

Bigger VRAM doesn’t always translate to a bigger performance. Especially in our tests which are very inference heavy. This doesn’t mean this card is DoA or something (pls don’t be, we need alternatives to the leather jacketed overlord).

There are many good takeaways. Such as AMD GPU software team are now finally getting their stuffs together. PyTorch for ROCm on Windows was one of the biggest to-do list. ROCm 7 is being faster than ROCm 6 (at least on RDNA4).

They also managed to lower the size of ROCm Docker image and PyTorch package.

Unfortunately, turns out it was due to they chose to drop RDNA2 from their supported GPUs. Sad.

AMD Radeon ROCm 7 supported GPUs

We still think there would be a better value for a bigger VRAM for different use cases. Probably in LLM finetuning, in 3D modelling/rendering, which we presume wouldn’t be easier to work around with CPU offload the way we tested MoE LLM inference above.

The post Our Experience with Asus AMD Radeon AI Pro R9700 Turbo appeared first on Efison Lisan Teknologi.

]]>