Why it matters
UDFs are essential when SQL isn't enough — parsing custom formats, calling external services, encoding domain logic. But they also have risks: bugs kill workers, slow UDFs slow every query using them.
The architecture
A simple UDF extends UDF class and implements evaluate() with typed arguments. Hive uses reflection to bind. The GenericUDF alternative is more flexible with complex or generic types.
UDAFs implement partial aggregation: process partial state per node, then merge partial states, then finalize. This lets aggregation parallelize.
UDTFs implement forward() to emit rows; used with LATERAL VIEW to explode arrays or maps.
How it works end to end
Deployment: package as JAR, upload to HDFS, ADD JAR in session, CREATE FUNCTION to register. Newer versions use permanent function registration in the metastore.
Performance: UDFs run once per row and cannot be vectorized. A slow UDF on billions of rows dominates query time. Consider whether the logic can be expressed in native SQL.
Isolation: UDFs run in the executor JVM. Bugs, memory leaks, or blocking I/O can affect other queries running on the same node.