r/elixir • u/ShininLotoen • 4d ago

Sending files over in a chat application

Hello, I'm working on a little chat application for fun as a learning experience. I just came across this roadblock which is, how do you send files over when you're messaging someone ? To be more specific, I am using a react frontend to send these files, and such files will be stored using aws s3, and I already have the waffle package to handle sorting these files into s3.

We already have a system for sending files using HTTP requests, which is by converting the files sent into the backend to a %Plug.Upload{} struct, but in the case of chatting to other users, HTTP won't cut it.

I know it's not good to use websockets as it's not the best at handling big chunks of data, so what would be the best method to send files from the React frontend, to the backend Elixir-wise and be able to send that information to another user ? Any help would be appreciated !

8 Upvotes

91% Upvoted

17 comments sorted by

10

u/pullmorematches 4d ago

Upload to s3/r2 or something and include the link in the message? could be a signed url if you are feeling fancy

7

u/NOLAnuffsaid 4d ago

This would be my approach.

For security reasons, never allow direct file sharing in your application.

1

u/BrotherManAndrew 3d ago

How should the frontend be able to upload directly to the s3, to me that seems like giving a flamethrower to a random person, 99 percent chance they don't burn a house down but still... you never know

Anyways let's assume the user can upload (somehow) and include the link, would the user have to first wait for what they sent to finish uploading and then they can get their link back to send the message or can they just immediately send the message and once the link actually points to the full attachment the media will be visible.

what would the system behind this whole process be? In what way will they upload and in what way will they get the link

thanks for the response

4

u/self 3d ago

first part of your question: the usual way is the frontend asks the backend for an s3 url, backend generates a digitally signed url (with a time limit) for http PUT or POST, frontend uploads to that. this way the s3 keys are not exposed.

7

u/ericls 4d ago

Send client a pre-signed URL. Let client do it

5

u/CompetitionDouble420 4d ago

aws s3 presign s3://<path_to_file> --expires-in <desired_ttl>

default ttl is 3600s (1hr) if not explicitly specified

2

u/Agile_Use_1768 4d ago
  1. Always send the message first (optimistically) with the promise of a future file attachment, probably by generating an attachment id and its presigned upload url.
  2. Notify that the file had finished uploading successfully with R2 or S3 bucket events hooks.
  3. Your ALB should fan-out this event and the Elixir pod hosting your Chat Room should pick it up and generate a presigned url to view the file, then broadcast it to room members through ws transport.

1

u/BrotherManAndrew 4d ago

Thank you but I have a small question,

Imagine the image was in the text so "balh blah blah image blah blah blha" what would the intermediary state of that image be?

- should it just be an id and once the resource is uploaded it will get the associated record and the rest is obvious,

  • should it only be set up when the file is uploaded?
It won't be like discord or telegram (?) or some other social media where the attachments are just at the bottom

What is the right way to go around this if you know

2

u/amzwC137 Alchemist ⚗️ 4d ago

Imagine the image was in the text so "balh blah blah image blah blah blha"

Interesting, now that you mention this, I don't know if I've ever seen this as possible. If you are sending an image it's always essentially as an attachment to the message. So I'm not sure that I can visualize the use case. (Outside a word/Google doc)

That said, I think the flow would be the same. When sending the message a reference/id to the image would occupy the image location in the message. Then once the message is retrievable on the recipient side, the message will render with the inlined image.

2

u/themeowbud 3d ago
  1. When client wants to upload a file, ask for presigned upload url to s3
  2. Client uploads the file
  3. Verify the file exists using s3 head object call so that the file actually exists and you dont pull the entire file into your server
  4. Send the message - the url to the object (presigned, almost always), and the associated text as a single message
  5. The client parses the message and displays in whatever way
  6. Optionally use blurhash while the image is being loaded

If you want the image in between text or something - thats on the backend and client to handle the positioning and rendering. The upload flow has nothing to do with it.

1

u/BrotherManAndrew 3d ago

Why should the URL be presigned, wouldn't it eventually expire? how is it different from just having a normal url

1

u/themeowbud 3d ago

It can be a normal url but that won’t be secure, a better practice. And expiry shouldn’t be a problem, set a reasonable TTL depending on your use case.

1

u/BrotherManAndrew 3d ago

Hm, How would a presigned url be safer?, chats can run over years so the ttl would have to be much longer because people often go back to times a lot earlier in chat, If the TTL is then longer then what would be the point of it being signed?

1

u/AndyDentPerth 2d ago

The presigned upload TTL can be small and is independent of the download URL TTL. In addition, the buckets are configured so there is never any public "list" capability exposed.

So, you have to know the exact download URL (embedded in chat) and can't go browsing to find other content.

1

u/amzwC137 Alchemist ⚗️ 4d ago

Interesting, everyone seems to have great ideas but mine was kinda dumb. That said, I don't know if my idea is the same.

Since you are using S3 already, couldn't you just upload the file to an S3 bucket, notify the recipient when the file is uploaded, then the recipient can retrieve the file on demand.

People mention signed urls, but wouldn't that be more important if you don't control both sides of the conversation?

2

u/Intcptr650 3d ago

What you’re describing is feasible but not recommended.

If you were to deploy this backend service in aws cloud you would incur the bandwidth cost associated with file uploads and downloads through your backend. Most cloud services I’ve seen allow a certain egress bandwidth and charge extra beyond a certain limit.

Presigned urls are recommended because they’re secure and offload the transfer to the user. For example you could just store the object location in user chat cache and ask your backend for a pre signed url whenever you wish to download the file. If the file is overwritten also this works. Also consider transient errors during download, limits on file sizes, retries. All this is better handled at the client level rather than on the server.

On the notification part you mentioned, it works the same way with pre signed urls. The backend generates a url that allows upload to s3, client uploads and then embeds the s3 path in the message and sends the message to backend. The recipient receives the packaged message and you can show a placeholder loading icon while the file is downloaded.