NVIDIA releases agent skills for Spark UDFs

NVIDIA has published a set of "cuDF skills for Apache Spark" that let coding agents rewrite user-defined functions (UDFs) into GPU-accelerated code. In a technical blog post, the company says the skills cover the full workflow, from testing and benchmarking through integration, and that the rewritten UDFs can make Spark jobs more than 22x faster.

The cuDF plugin for Spark already accelerates SQL and DataFrame queries without code changes. It can do this because Spark's declarative interface lets the plugin swap in GPU implementations of each operation. UDFs are the exception. They contain arbitrary user code that the engine cannot inspect, so the plugin has to convert data from columnar to row-wise format and run the UDF row by row on the CPU. That means a round-trip data transfer, and the UDF itself gets no GPU speedup.

NVIDIA describes three ways to rewrite a UDF, in order of complexity:

• SQL: replace simple logic with native Spark SQL expressions that the plugin can map to the GPU.
• cuDF: implement the RapidsUDF interface's evaluateColumnar method so the UDF works directly on GPU-resident columns. NVIDIA says this suits most cases, including string parsing and nested types.
• CUDA: call custom kernels through JNI when logic is too complex or performance-critical for cuDF.

NVIDIA says hand-written rewrites take a lot of engineering effort: picking the right APIs, checking that results match, and measuring performance. It argues that agents with guardrails such as executable tests and benchmarks can handle this work in minutes.

The available excerpt does not say which agents or models the skills support, how they are packaged, or which workloads produced the 22x figure.