Deploying FuriosaAI RNGD for Korean Cloud Inference

The TCP architecture drives inference efficiency

TSMC fabricates the RNGD NPU on a 5nm process node. The Tensor Contraction Processor (TCP) architecture treats tensor contraction as a native hardware primitive to minimize data movement. This design delivers 512 TFLOPS for FP8 workloads, 256 TFLOPS for BF16, and 1024 TOPS for INT4. The chip integrates two HBM3 modules that provide 1.5 TB/s of aggregate memory bandwidth and 48GB of capacity. It operates at a 1.0 GHz clock frequency and uses a PCIe Gen5 x16 interface. Processing Elements (PEs) can function independently or fuse to act as a single large processing element. Each PE contains a CPU core for scalar processing and a 256 KB L2 cache. The architecture uses a circuit-switching fetch network to ensure flexible data access and allows extensive data reuse across compute units. Within each Tensor Unit (TU), the 32 MB of SRAM capacity supports high-throughput execution. The Contraction Engine (CE) uses eight dot product units to perform dot products and accumulate results. The Vector Engine (VE) handles higher precision tasks like FP32 and INT32. LG AI Research found that RNGD delivers 2.25x better performance per watt for LLMs compared to a GPU-based solution. LG AI Research reported that an RNGD-powered rack generates 3.75x more tokens for EXAONE models than a GPU rack operating within the same power constraints because of the higher compute density. Compared to the NVIDIA H100, which has a 350W TDP and 80GB of memory, the RNGD card uses only 180W of power. However, the 48GB HBM3 memory capacity limits the size of models that can fit on a single card.

Accessing RNGD via Samsung SDS

Samsung SDS hosts RNGD accelerators at its Dongtan IDC in Gyeonggi Province. This facility opened in March 2023 and has six floors with four server rooms. The data center uses natural air cooling to maintain a PUE of 1.12. Samsung SDS provides NPU-as-a-Service (NPUaaS) on the Samsung Cloud Platform to provide efficient compute at scale. The service uses the NXT RNGD server, which can host up to eight accelerators per server. This configuration provides a total of 384 GB of HBM3 memory and 12 TB/s of memory bandwidth. The hardware uses Single Root I/O Virtualization (SR-IOV) to create up to 8 independent NPU instances. These virtual instances use dedicated compute cores and private memory bandwidth to ensure deterministic latency for co-located workloads. Samsung SDS provides on-demand configurations of 1, 2, 4, or 8 cards. The deployment targets high inference capacity per rack for agentic AI workloads. If you already manage Kubernetes clusters, the integration remains straightforward.

Setting up the Python SDK and LLM serving

Setting up the environment requires Python 3.8 or higher. Most developers use Conda to create isolated environments and prevent dependency conflicts. After activating the environment, users must install the required kernel driver, firmware, and runtime library before installing the Python SDK via pip install furiosa-sdk. The software stack contains the Furiosa-LLM runtime and supports the open-source vLLM framework. The Python SDK provides an inference API, a quantization API, a command line tool, and a server program. Developers deploy models using pre-compiled FXB bundles from Hugging Face. For instance, running the Qwen3-32B model with FP8 precision requires the command furiosa-llm serve furiosa-ai/Qwen3-32B-FP8. For hybrid reasoning models like Qwen3, the qwen3reasoning parser returns the chain of thought in a separate field. Users can also build custom bundles with the fxb build command. The SDK includes a native just-in-time compiler for PyTorch. The software supports models such as EXAONE-4.0-32B-FP8 and Llama-3.3-70B-Instruct. How will the scalability of the PCIe Gen5 interface compare to proprietary interconnects as models grow even larger?