Implement sticker packs

[rocurley]

I'd like to add support for sticker packs to Riot. Since this would involve writing a lot of code, I wrote this little document up so we could discuss it before I've spent a long time on it. That said, while I'd love to have help, I'm happy to implement this myself.

Why sticker packs?

Most of my friends use Telegram as their chat client of choice. This isn't ideal for several reasons, and I'd like to get them to switch. It may seem silly, but I'm pretty sure the biggest lock-in factor there is stickers. Telegram's sticker system is really slick, and using anything without it or something similar feels like a real downgrade. Relatedly, I've framed most of this in comparison to Telegram's sticker system.

What makes sticker packs good?

I'd claim that the most important features are:

• User creatable: This is nice on several levels. It saves work on the dev side, obviously. But more than that, this lets people make stickers for their specific interests.

• Easily sendable: Stickers won't get used if it's not convenient to do so.

• Easily copyable: Almost all the stickers I use, I have because I saw one I liked, and I added it to my collection. Telegram makes this process trivial.

• Grouped: Part of the reason stickers spread so effectively is that if I see a sticker I like and add it to my collection, I get a bunch of other stickers along with it that I will likely enjoy.

The ideal outcome is an environment where stickers can be easily
created, and easily spread between people when they chat with each
other.

What are features we could do without?

I'm not super confident here, but:

• Updating packs. When the creator of a sticker pack adds new stickers to it, that change could be propagated out to users who have the pack, or not. I'm inclined towards not supporting that, mostly on implementation difficulty grounds. One could imagine having sticker packs label themselves as replacements for other ones, and make it easy to replace the old one if the user adds the new sticker pack, but that probably doesn’t need to be in the initial version.

Implementation

I’m proposing a specific implementation here as a proof of concept, but I’d be happy to see it improved. Mostly I’d like to get some discussion about this so I don’t spend a long time implementing this and then have it rejected.

The design goals with the implementation were:

• Work entirely within within matrix: no external servers.

• Avoid changes to the matrix protocol.

• Avoid requiring the homeserver to know about stickers.

• Avoid nonstandard messages.

This design achieves the first three. To allow subscribing to sticker packs, the fourth is compromised by adding a custom field to the image message when stickers are sent.

A sticker pack is an archive containing one or more images (stickers), and a metadata file. The metadata would associate each sticker with zero or more emoji, and contain the name of the sticker pack. It would also contain whatever information I forgot was important. Since it's likely we won't know ahead of time what information is important, we can version the spec and save the version in the metadata file. It’s unclear what file formats should be supported: that probably depends on how inline images shakes out (although this can be done with the standard image sharing until then).

To publish a sticker pack, one would:

• Create this archive (ideally assisted by the client, but it could start out as a manual process).

• Upload the sticker pack to your homeserver. The resulting MCX URI enables other users to subscribe (a misleading term, since it's impossible to update a sticker pack) to the sticker pack.

To subscribe to a sticker pack, the client would:

• Download the pack using the pack's URI.

• Save the URI in the client config. This makes subscribing to a sticker pack a a per-user, not per-device affair.

When using the pack, the client would:

• Check to make sure all the sticker packs in the client-config exist locally. This includes deleting sticker packs that have been unsubscribed from.

• Allow the user to easily browse and send stickers in the sticker packs.

• When sending a sticker, include in the message the sticker pack URI. This allows other users to preview and subscribe to other sticker packs.

• (This can be added late) If the user starts a message with an emoji, show all stickers indexed by that emoji.

Stages

I think the natural way to break this down into sub-tasks, each of which
can be implemented separately, is:

1. Implement sending (manually created) sticker packs from local storage. This is somewhat similar to Gregoor’s proposal for issue 2648. I’d be happy to coordinate with them on this.

2. (This doesn’t block anything else, it just has to come after 1) Implement a UI for creating sticker packs.

3. Synchronize a user’s sticker packs via the client-config.

4. When using a sticker from a pack, attach the link to the sticker pack.

5. Make a UI for subscribing to sticker packs you’ve seen.

[tessgadwa]

Let's do it.

See my comments at https://matrix.to/#/!blHeuKmYNNfIqDlWGO:matrix.org/$14844481771476529VOaQj:matrix.org

[rocurley]

Some details that tessgadwa and I hashed out over chat (it's all at the link above, so you can just read it there if you prefer).
Stickers should be 512x(<=512) vice versa (one dimension 512, one <=512) pngs, or vector graphics (svgs?).
Stickers can be associated with emoji. This implicitly requires the ability to recognize emoji. it seems simplest to piggyback off of the Emoji One conversion process (where if you get an emoji and nothing else in a message, it turns it into a bigger emoji).
We also discussed making some number of default sticker packs, which tessgadwa is interested in doing (in addition to UI work for steps 2 and 5). Stickerpacks of about 12 stickers, ideally including ones that convey: happy, bored, sad, angry, curious, celebratory, congratulatory, amused. (This was pretty off the cuff though).

[rocurley]

Some more discussion here. We're pretty much set for stage 1 as described above.

It was suggested that we find an existing sticker pack format and just use it. The only such thing I could find was apple's here. I'm sort of disinclined from using it, because it supports a bunch of features we probably don't want (animated stickers, for example), as well as not supporting some features it should (emoji). I think a simple json format along the lines of:

{
    "name" : "Goats with sunglasses"
    "author" : "Nunya Business"
    "stickerpack spec version" : 1.0
    "stickers" [
        {
            "file" : "evilGoat.svg"
            "emoji" : "😈🔥"
        },
        {
            "file" : "similingGoat.png"
            "emoji" : "😊"
        },
        {
            "file" : "nondescriptGoat.png"
            "emoji" : ""
        }
    ]
}

would be a lot easier.

We discussed where to store packs, with the tentative plan to use mxc for now. Other options include IPFS, some website, or torrent hashes. We probably want to have an external website for searching for sticker packs as well at some point.

[tessgadwa]

This works for me.

[ara4n]

how are the emoji fields meant to be used here?

[rocurley]

If the user starts a message with an emoji, show all stickers indexed by that emoji. Since stickers are used more or less in the same context as emoji, showing a relevant sticker when you type an emoji can be pretty useful

[tessgadwa]

I was also noticing that when I used the built-on Riot emojii, I only get 8 emojii showing up. Is this standard?

[tessgadwa]

Potential artist to contact suggested: http://www.deviantart.com/art/Phil-The-Owl-629856759

[tessgadwa]

Good discussion about in Riot main room today about image uploading vs sticker packs.

Goal is to allow users to access a common set of sticker packs direct from the app without having to upload their own each time from the hard drive.

[tessgadwa]

Another big goal: Create a way for artists to upload/share stickers under a CC license for inclusion in Riot.

[rocurley]

Currently thinking we should just base64 encode the images and stick them in the json file directly. It's a bit inefficient (4/3), but that way we don't have to add a dependency on some tar library or store every sticker separately on the server.

[ara4n]

@rocurley the disadvantage of doing that is that you don't end up with a canonical URL for the individual images, which will encourage clients to go and take the base64 out of the JSON and keep uploading them in duplicate to their HSes. Ideally the HS should deduplicate the content, but currently synapse doesn't sadly. Whereas if you reference the actual bits of artwork as separate mxc:// URLs, they will automatically get deduplicated. And clients won't have to mess around re-uploading when they send 'em?

[rocurley]

@ara4n That's definitely a benefit of storing them separately. My main concern there is individual stickers getting deleted. If you just have one file, that's less likely to happen. I still don't have a great model for how likely files are to be deleted though.

[rocurley]

Also worth figuring out: How do we cache stickerpacks locally? I was planning on fetching them from the homeserver at startup every time for now, but I don't love that option. I looked into localstorage, but 5Mb isn't very much.

[ara4n]

I'd just cache stickers using normal HTTP caching in the client (at least for riot-web); it should work fine :) In terms of content being deleted - there's always going to be a risk that the source content is deleted, whether it's a master copy or one of a million duplicates. For now I'd completely ignore it; practically speaking folks in Matrix hardly ever expire MXC content (and even if they do, it's only their remote caches), and in future switching to something like ipfs will make it go away anyway.