Arbitrary alphabet and radix support

[anitgandhi]

Description of Enhancement

Expanding on #1 , in theory, this package should support arbitrary alphabets as well. It will require either major changes to math/big or a new sub-package.

This may or may not have to include non-ASCII characters.

Further, there is a chance it will require a breaking change in the interface, if this alphabet input is added to NewCipher. Alternatively, a new function on Cipher called SetAlphabet could be added.

func NewCipher(radix int, maxTLen int, key []byte, tweak []byte) (Cipher, error) {
	...
	// defaults
    f.alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    f.radix = 62
}

func (f Cipher) SetAlphabet(alphabet string, radix int) {
	// error checking on alphabet and radix
	// ex: alphabet can't be "12345" and radix be 9
	...

    f.alphabet = alphabet
    f.radix = radix
}

[fmpwizard]

Two questions:

1. Did you already submit/propose the changes to the Go project? (have a link?)
2. I have been looking for something like this for a while now, but the string I have to encrypt, besides having 0-9, a-z, A-Z, also could have things like / % & and a few other characters, would they fit into the idea behind this ticket of alternative alphabets ?

Thanks a lot for making this project public!

[anitgandhi]

1. I haven't yet, thanks for the reminder 👍
2. Yup the arbitrary alphabet support would be perfect for things like / % &. I haven't gotten a chance to take a deeper dive into how it would work in detail; in one sense it's easiest to convince Golang maintainers to add this support, but on the flip side that would require the newest version of Golang, rather than being self-contained.

[fmpwizard]

It would be great if the change wouldn't require using the latest Go version (specially using tip), but have that as a planned migration. What I mean is, add it as a subpackage, so this can be used soon, and from the start making it clear that once Go updates math/big , then the subpackage won't be needed any more

[anitgandhi]

Agreed - I'd prefer it to not be dependent on the Go version.

I like the migration option long term, but as it is that's dependent on upstream Go.

I tested the base62 support ( #1 ) in a branch by copying the standard math/big into fpe/big and making the changes there, and it worked fine. However, I'm a little uncomfortable importing an entire existing library as a sub-package like that and then modifying; potential license conflicts and other issues.

Further, math/big has a lot of other functions that aren't relevant to this package.

I think for now what I will do is use only the parts of math/big and potentially re-write as well. This may provide the opportunity for further performance optimizations as well :)

[anitgandhi]

@fmpwizard discussion is here: golang/go#21558

[fmpwizard]

I see that the linked ticket was closed, but they didn't go with the change that would allow arbitrary alphabets, right?

[anitgandhi]

@fmpwizard that is correct, they added base62 support which basically fixes #1 , which is nice; but not full arbitrary alphabets. The Go team's comments on the linked issue explains the reasoning behind that.

I do still want to add arbitrary alphabet support, especially seeing your use case above. I may just have to add separate func (z *big.Int) TextWithCharset(charset string) and func (z *Int) SetStringWithCharset(s string, charset string) that only deal with fpe related use-cases. As the Go team mentioned, it would introduce complexities with encoding/normalization that is beyond the scope of the current need. It may end up being as simple as copying the old internal digits function from <Go1.5 and modifying it appropriately; it may not be.

If you have any thoughts on how to do this effectively, please let me know!

[fmpwizard]

@anitgandhi thanks for following up on this topic. I can more or less follow the ideas around implementing this, but not enough to propose an actual changeset, let alone doing it effectively.

[fmpwizard]

friendly ping

[anitgandhi]

Sadly I haven't had the bandwidth to work on this issue 😢

As a note to my future self (or anyone who wants to take this up), implementing this will have to limit to ASCII strings so the []byte <--> string conversions will hold up. UTF-8 and other aspects would complicate it.