Determines whether FGL is using ubdds, hons-aigs, or aignet literals as its Boolean function representation.
This is encoded using the numbers 0, 1, 2 so that it can be packed into a bfrstate object efficiently. 0 means aignet, 1 means UBDDs, and 2 means hons-AIGs. But please use bfr-mode-case instead of explicitly checking for these values.