Support gallery messages

[bxdxnn]

Content

Support sending and rendering gallery messages (a message with multiple photos)

Motivation and context

Screenshots / GIFs

Tests

• Test that sending works
• Test that rendering works

Tested devices

• Physical
• Emulator
• OS version(s):

Checklist

• This PR was made with the help of AI:
  • Yes. In this case, please request a review by Copilot.
  • No.
• Changes have been tested on an Android device or Android emulator with API 24
• UI change has been tested on both light and dark themes
• Accessibility has been taken into account. See https://github.com/element-hq/element-x-android/blob/develop/CONTRIBUTING.md#accessibility
• Pull request is based on the develop branch
• Pull request title will be used in the release note, it clearly defines what will change for the user
• Pull request includes screenshots or videos if containing UI changes
• You've made a self review of your PR

[github-actions]

Thank you for your contribution! Here are a few things to check in the PR to ensure it's reviewed as quickly as possible:

• If your pull request adds a feature or modifies the UI, this should have an equivalent pull request in the Element X iOS repo unless it only affects an Android-only behaviour or is behind a disabled feature flag, since we need parity in both clients to consider a feature done. It will also need to be approved by our product and design teams before being merged, so it's usually a good idea to discuss the changes in a Github issue first and then start working on them once the approach has been validated.
• Your branch should be based on origin/develop, at least when it was created.
• The title of the PR will be used for release notes, so it needs to describe the change visible to the user.
• The test pass locally running ./gradlew test.
• The code quality check suite pass locally running ./gradlew runQualityChecks.
• If you modified anything related to the UI, including previews, you'll have to run the Record screenshots GH action in your forked repo: that will generate compatible new screenshots. However, given Github Actions limitations, it will prevent the CI from running temporarily, until you upload a new commit after that one. To do so, just pull the latest changes and push an empty commit.

[bxdxnn]

@americanrefugee please review the design. Particularly:

• How to make a better border between tiles?
• What should be the border radius of the message? You can see on the screenshot, the bottom-right corner that "connects" messages doesn't look good.
• More complex, but please test the logic of handling landscape/portrait messages in a gallery and if any changes are needed.
• Tapping on a picture in the gallery will allow viewing/swiping only the pictures of the gallery, but in a non-gallery message, you can see all pictures.

[americanrefugee]

I think we're are already working on this feature for iOS. cc @Velin92

[americanrefugee]

@bmarty we should coordinate to make sure the same thing is applied to Android.

[mxandreas]

How has been this accomplished? To make it work cross-platform and interoperable with other clients, it needs to use a MSC like this matrix-org/matrix-spec-proposals#4274 ? Is this the intention?

Also, this clearly needs support on all 3 platforms Web, iOS and Android before it can be generally released.

[bxdxnn]

How has been this accomplished? To make it work cross-platform and interoperable with other clients, it needs to use a MSC like this matrix-org/matrix-spec-proposals#4274 ? Is this the intention?

Also, this clearly needs support on all 3 platforms Web, iOS and Android before it can be generally released.

SDK already supports gallery messages:
matrix-org/matrix-rust-sdk#4977
matrix-org/matrix-rust-sdk#5125

[mxandreas]

SDK already supports gallery messages:

Ok, that clarifies the technical approach. My suggestion is that if we want to move forward, we need to first confirm, if there is capacity to implement this also for Web. If Rust SDK supports it, I am guessing it may not be such a lot of effort for iOS. However, Web is not using Rust SDK, thus needs to be checked first.

If all of that looks feasible then we still need to figure out the designs, but I think we're able to do that if there if there is confirmation it to be done on all 3 platforms.

[bxdxnn]

If all of that looks feasible then we still need to figure out the designs, but I think we're able to do that if there if there is confirmation it to be done on all 3 platforms.

I was able to do a POC Web implementation, do I need an iOS PR too for this feature to proceed with the design?

[mxandreas]

I was able to do a POC Web implementation, do I need an iOS PR too for this feature to proceed with the design?

It is sufficient if we know that you want and can do it on iOS. Which I understand is the case, correct? @americanrefugee can you look into the designs?

A couple of notes:

1. The MSC is built so that you can have one caption per gallery.
2. The MSC works for images/videos but also files (attachments). It would be great to support it for all types - there's less confusion, e.g. another client may send you a gallery of files, and if you do not support rendering it (while we support rendering images), it is confusing.
3. Currently if you open an image/video or an attachment from timeline - you can swipe left/right to go to the previous/next in the room - if now the item is part of a gallery, should such browsing be limited to the gallery items only, or you would be able to "exit" the gallery and see items which are before and after? I think yes.

[americanrefugee]

Here's some direction for the media gallery in a timeline...

THE EXAMPLE BELOW IS ONLY FOR ANDROID

• For iOS, the logic is exactly the same, but the fonts are specific to iOS
• This is NOT for Web (yet to be designed)

Context

Here's an example of multiple media items uploaded to a timeline, with and without a caption:

Gallery item preview

Here's an example of images or video gallery items, including overflow, in Light and Dark themes:

Full gallery

And here's an example of the various gallery groupings with 1, 2, 3, 4, 5, and 5+ items:

• Spacing between items = 4px
• Corner radius for the whole group = 6px

[americanrefugee]

We will also need a version of a gallery message in the timeline for files.

Since files may or may not contain a preview, for this version we should show a list:

The logic is as follows:

• If the user uploads files that all have a preview (such as a bunch of PDFs), then we can still show them like you've already built for images and videos.
• But if the user uploads a mixture of any files that do not have a preview, then all the files are rendered together as a list (shown above).
  • The icon for any document (PDF, ZIP, DOC, etc.) is "Attachment" (paperclip)
  • Audio files show a play button, length, and sound bars
  • Images and videos show a thumbnail of the image, file name, file type, and size.

When replying to a gallery, the bubbles should look like this:

Does this make sense?

[americanrefugee]

I accidentally added images above for iOS. I just updated them to show Android. Sorry!

[mxandreas]

If the user uploads files that all have a preview (such as a bunch of PDFs), then we can still show them like you've already built for images and videos.

@americanrefugee At least currently we do not render any preview for a PDF. Let's keep it the same way now.

[bxdxnn]

@americanrefugee

I've implemented the designs, but can you send/tell more precise style attributes for the designs you've sent if possible? Particularly, I want to know the size of the video icon, the gradient attributes, and the timestamp position attributes.

And here's an example of the various gallery groupings with 1, 2, 3, 4, 5, and 5+ items:

FYI, this example is too simple as it doesn't take into account landscape/vertical mixed media grid handling. I've left the current grid compaction logic without changes, as it should already shows landscape-only galleries like in the example.

• Audio files show a play button, length, and sound bars

The waveform will need to be generated each time you encounter an audio file in a gallery message, which I believe can lead to big performance issues as the audio file will need to be downloaded before generating the waveform. Imagine if someone would send a 1 GB audio file, it'd take a while and might cost a lot if the user is on a metered connection.
Alternatively, we can add the pre-generated waveform for each audio file in the gallery, the same way how we're doing it right now with the voice messages. But I guess that wasn't defined in the MSC and additionally we'll need SDK support too.
Do you really want to basically convert audio files to voice messages? Besides the technical reasons, this is inconsistent with how audio files are normally displayed, and it will also make it impossible to save/share the audio file, unless we'll make a bottom sheet for the attacment rows.

[americanrefugee]

this example is too simple as it doesn't take into account landscape/vertical mixed media grid handling

Regardless of the original images's aspect ratio (landscape/vertical/square), the image previews should be displayed in that grid. Each image should be centered and fill the box. So if you have a tall image, it would appear as a horizontal preview if in the top position. In other words, the grid does not reflect the actual image aspect ratios.

The reason is because 1.) we want to keep this super simple and 2.) we're trying to conserve vertical space in the timeline. A gallery shouldn't take up the whole screen.

Do you really want to basically convert audio files to voice messages?

No, you're right. In the timeline an audio message should display an audio icon, file name, file type, and size - NOT the play button and sound waves. I've updated the design, see below:

I want to know the size of the video icon, the gradient attributes, and the timestamp position attributes.

• Icon = 16x16, positioned 8px from left and bottom edges
• Timestamp = font/body/sm/medium, positioned 8px from right and bottom edges
• Single video preview = Gradient from 0% opacity to 100% color/bg/canvas/default (top to bottom)
• Overflow preview (with "+X") = color/gray/100 at 70% opacity (no gradient)

Let me know if you need any other values.

[bxdxnn]

Let me know if you need any other values.

Please tell the details (like size) for the tiles in the "Full gallery" example.

Each image should be centered and fill the box

FYI, I'll assume that if the image is narrow it should be zoom-in-to-fit, and if too tall/wide, crop-to-fit, both go to the center of the image.

[americanrefugee]

Please tell the details (like size) for the tiles in the "Full gallery" example.

It depends on the size of the message bubble. Let's say that the overall message bubble width is 292px...

• There is an 8px margin on the left / right side between the message bubble edge and its contents
• That leaves 276px
• There is a 4px gutter between image previews
• If the bottom row is 1 image, then the height is 50% of the width
• If the bottom row is 2 square images, and with a gutter of 4px, they squares would be 136x136
  • If the top row is also 2 squares, then they would be the same size for a total of 136 + 4 + 136 =276
  • If the top row is 1 rectangle, then it would be 276w x 136h
• If the bottom row is 3 square images, and with a 4px gutter, the squares would be 89.3x89x3
  • If the top row is 2 squares, then they are 136x136
  • f the top row is 1 rectangle, then it would be 276w x 136h

Or, another way to describe the sizing...

• All square images are exactly square with a 4px gutter between them
• All rectangular images are the same size as 2 squares width and 1 square hight
• There is always a 4px gutter between rows

Does that help?

[bmarty]

We will merge this WIP feature (FF is disabled) to land the code to develop. I have created issues to track the remaining work, and they may be other issues to fix.

Thanks for the initial work @bxdxnn !

[ganfra]

Some more remarks after a quick look

[bmarty]

Merging on my branch so that I can record screenshots.