Representing Unordered Data Using Complex-Weighted Multiset Automata
Justin DeBenedetto,u00a0David Chiang
Unordered, variable-sized inputs arise in many settings acrossmultiple fields. The ability for set- and multiset-oriented neuralnetworks to handle this type of input has been the focus of muchwork in recent years. We propose to represent multisets usingcomplex-weighted multiset automata and show how themultiset representations of certain existing neural architecturescan be viewed as special cases of ours. Namely, (1) we provide a newtheoretical and intuitive justification for the Transformer modelu2019srepresentation of positions using sinusoidal functions, and (2) weextend the DeepSets model to use complex numbers, enabling it tooutperform the existing model on an extension of one of their tasks.
