Neural Power Units
Niklas Heim,Tomas Pevny,Vasek Smidl
Conventional Neural Networks can approximate simple arithmetic operations, butfail to generalize beyond the range of numbers that were seen during training.Neural Arithmetic Units aim to overcome this difficulty, but current arithmeticunits are either limited to operate on positive numbers or can only represent asubset of arithmetic operations. We introduce the Neural Power Unit (NPU) thatoperates on the full domain of real numbers and is capable of learning arbitrarypower functions in a single layer. The NPU thus fixes the shortcomings of existingarithmetic units and extends their expressivity. We achieve this by using complexarithmetic without requiring a conversion of the network to complex numbers. Asimplification of the unit to the RealNPU yields a highly transparent model. Weshow that the NPUs outperform their competitors in terms of accuracy and sparsityon artificial arithmetic datasets, and that the RealNPU can discover the governingequations of a dynamical system only from data.
