How much does it cost twitch to store content on firebase?

I doubt Twitch uses FireBase.

First of all, I’ve tried to check Twitch open Job positions for Firebase, so I Googled “site:www.twitch.tv/jobs/ firebase” and got zero results.

Secondly, when a company infrastructure grows to the size of Twitch, it doesn’t really make sense to use specific proprietary technology like Firebase. The risks of getting technical problems with such technology are too high.

Thirdly, it doesn’t make sense financially to store video content on Firebase. Let’s check the numbers, data storage in Firebase cost is $0.18/GiB per month.

Today I wrote an article about Twitch expenses on video streaming and found out that Twitch streamed in August 2019 roughly 1256 Petabytes of content: How much does it cost to maintain Twitch infrastructure?

If we only assume that 20% of that content is saved, and Firebase makes a big 50% discount, that is 251 Petabytes x $0.09/GiB = $23.5 million USD per August only. Doesn’t sound like a reasonable expense.

I’ve found traces online that Twitch used Firebase sometime ago, and I guess they haven’t used it for video storage.

If you ask this question trying to find out where to store content, this article should give some information: Paying hefty bills for Amazon AWS traffic and storage? Why not to save 2x+ times

First, when talking about databases in the Firebase suite, I’m only going to talk about Firestore, and not RTDB (Realtime Database, which was traditionally known as just Firebase). Firebase now refers to an entire suite of services, which includes Firestore, RTDB, file storage, authentication, etc. RTDB is most likely going to be phased out and Google is clearly putting their effort behind Firestore.

Second, there is no manual sharding in Firestore, like there was in RTDB. Sharding is handled automatically by Firestore, so ignore the other answers about that.

Third, you, the customer, own the data in your Firestore database, not Google, so ignore the other answers about that also. If Google, and not the customer, owned that data, Firebase would have virtually no users. And, also contrary to the other answers, you can export data from Firestore. If customers weren’t allowed to export their data, who would ever use it?

Fourth, the suggestion to use SQL technology on your own physical servers (ones that you physically own) is not something that I think belongs in an answer to this question. SQL vs NoSQL and cloud vs in-house are decisions that you should have already made if you’re asking about a specific serverless provider. Clearly (at least apparently), you’ve decided on serverless NoSQL, which is definitely the most popular choice today. And between the two most popular providers, AWS and Firebase, they are designed to handle virtually any workload you throw at it. The most highly-trafficked applications in the world run on NoSQL serverless, like the Amazon marketplace, for example (AWS, obviously). If AWS can handle it, Firebase can handle it too. Google would never design its NoSQL serverless to max out at a certain number of users or level of traffic; it’s designed to scale virtually infinitely.

The only caveat to Firestore (and NoSQL serverless in general, really) is dollar cost—right now in 2019, anyway. Firestore was considerably more expensive only a year ago than it is today, think about that. I think NoSQL serverless will undoubtedly continue to become cheaper, including Firestore. But, right now, with a relatively large userbase, NoSQL serverless, like Firestore, can get expensive. But so can doing it all in-house. An in-house solution still comes with bandwidth cost, in addition to hardware and human cost. Hiring qualified backend engineers to build and maintain an in-house setup is not cheap (hundreds of thousands of dollars a year). Firestore (and DynamoDB) have excellent price calculators to help you predict your costs and you may be surprised to see how cheap it is with a modestly-sized userbase, even with 100k users.

This is a question I get from a lot of clients as many of our products rely heavily on Firebase. The answer is very dependant on your application but in almost all cases this will never be a problem.

Where do the costs occur?

If you look at the Firebase pricing plan you can see that data is the main constraint for the pricing brackets. When you move up a tier you get various increases in data allowances as well as more database operations. This means you need to look at what your app does to see the impact of more users.

What does my app do?

This is the next question you need to ask yourself. Many apps will be dealing only in text strings meaning the data amount is very small. Images would be a step up from this and videos another jump in data usage. You need to think about how often your app stores this data. It might be that you only upload a single image for each user or it could be that you send multiple images while using. In both these cases the data usage will vary hugely.

Before worrying about how you will cope with 1M users think about how the app will fare with 1–10k users. In Firebase’s case you will probably be on the free tier (as long as you are not guzzling data) up to 100k users.

Think about this for a moment. You will have 100k users on your app. Even if you only add advertising you should be able to make the $25 a month to cover the Firebase FLAME costs.

The motto I use with my clients is this:

By the time you need to start worrying about backend costs for your app you can stop worrying about backend costs for your app

Once you reach the amount of users to requiring you to start payment you will have a successful app with a big user base which can be leveraged to start paying you back.

Should I use Firebase?

This is the final question you should ask yourself. Up until now this question could have been about one of the many different backends on offer.

If you are unsure I would be inclined to say: YES use Firebase!!

They are a great company and offer a fantastic product. Being owned by Google and having recently updated all their code they are unlikely to close down (RIP Parse). They also provide extensive and clear documentation and great support. There is a huge online community meaning sites like this and Stackoverflow have a huge number of questions to help you.

It is worth looking at your app though. Firebase does have some disadvantages:

  1. There is no easy way of querying data
  2. There is no way of executing code on the server

This is what should be researched when looking at whether Firebase is the right backend for you and even these can be mainly mitigated by using a separate server to control these tasks.

Conclusion: Unless your app is very data intensive then you won’t meet any of the data caps. Firebase will help you get your app online quickly and cheaply before scaling excellently once the users start pushing the upper limits.

NOTE: I would never recommend someone building their own backend when starting a new app. It might be that you have the skills to do it but in the long run it will cost you more (in your valuable time) than using another provider. Once your app is a great success and you are paying monthly fees to a backend then you can invest the money in creating a custom backend which suits your data type. Creating it yourself also requires a huge amount of invested time. If your app fails then this time is completely wasted.

No, for enterprise-grade application, firebase is not a good choice and not scalable as well.

Firebase is good choice till it’s not your primary database and you use it for the limited purpose where it shines.

Here is the list of limitations so you can decide on yourself.

  1. Big limitations written on their own documents:
    1. Firebase itself says that “Complex, hierarchical data is harder to organize at scale.” (The same can be referred on their doc here.)
    2. You can only sort or filter on a property, not sort and filter on a property, in a single query. (Reference)
  2. Firebase Real-time Database queries do not support WHERE clause.
  3. The major disadvantage is we can not query for more than one key/id at a time.
  4. Firebase can only sort ascending, not descending.
  5. Different combinations of select query criteria requires multiple times data duplication.
  6. One would never be able to filter one's data, this is a big pain at this stage.
  7. IP restriction is not supported.
  8. Fetching data of multiple users is not supported.
  9. Database structure needs to be changed on every little requirement change. Eg. If we want to change sorting order then existing any DB structure will be of no use.
  10. Firebase Real-time Database is built on a funda of Data Duplication (Denormalization) and due to this, it keeps increasing the overhead/load on clients and servers for keeping all the copies of data updated whenever there is a change in data values.
  11. child_added listener fails to provide the previous record Id when records are being inserted back to back (e.g. chat messages).
  12. Certain query options can't be combined. Because of this limitation, it forces us to either sort the documents by date or filter them with the user's search query on the database side and perform the other action on the client side.
  13. Complex queries are impossible. For instance, we want to build a Slack-like app: we cannot count unread messages, even if we have the proper structure that would allow doing so (eg: a read = <true|false> tag). The hack is to count all unread messages client-side, after having retrieved the whole dataset. It's a huge performance issue. It will increase the usage cost at user-end due to the high amount of internet consumption.
  14. As query filters are not supported, the large amount of data transmission occurs over the network for all the clients and servers communicating with database.
  15. With Firebase, we can not paginate because we can not get query array length, we can not paginate ordered arrays, etc.
  16. It has disadvantages if we are managing large amounts of highly structured data. With a regular relational database we can write powerful queries in SQL to quickly get the data we looking for - there is no equivalent with Firebase.
  17. We can not use ORM (Object-relational mapping) with Firebase to simplify data handling.
  18. As parent-child relation not supported, dealing with relations with Firebase is a pain. e.g. An user belongs to a group, and a group has users.
  19. With embedded platforms, REST API’s of Firebase cannot be implemented quickly.
  20. Firebase does not support simple Geo queries.
  21. Firebase is also not ideal if the application requires continuous processing on server side (especially when data has to be analyzed before there is some display to users real-time based on the result).
  22. Firebase definitely doesn’t support transactions.
  23. As all clients can connect directly with the database and can update data, all the clients become responsible for keeping data consistent which sometimes end-up with uncontrollable system behavior.
  24. Without a back end, supporting Web, iOS, and Android means rewriting business logic and data transformation numerous times.
  25. Using Firebase means that all the server logic is now running right in web or mobile client.
  26. In most apps, we have to send welcome emails, process images (avatars, etc), deal with payments, and build our business-core features. We really don't want all those to be done on the client, as this can range from impossible to dangerous for our business.
  27. If we think about distribution: any database logic change results in client app updates. How do we deal with clients who didn't update? Is it a correct thing to do for our users to deactivate older clients to force an update?
  28. With Firebase, it is impossible to expose API for our product.
  29. MIS (or any kind of) report generation becomes very difficult with Firebase database as there is no support of querying on multiple tables.
  30. No reliable tools are available to visualize and manage data. (e.g. SQL Server Management Studio, MySQL Workbench, etc.)
  31. We don't have root access to the location where our data is stored.
  32. No on-premise installation.
  33. Reporting tools are not strong enough in Firebase.
  34. Firebase users are vendor-locked.
  35. We can't shop around for which hosting provider to use.
  36. No case studies of enterprise level usage of Firestore.
  37. Firebase is a mobile-first offering. It was designed for mobile apps (Android, iOS, and web where sensible), and favors projects with a focus on mobile usage.
  38. Cost: Paying $100 per month to Firebase for something we can run on a $5 DigitalOcean droplet is something that gets us to think twice when dealing with server-less code.
  39. It's harder to migrate to a different service.
  40. There is no development roadmap for adding new features/capabilities in the future.
  41. We have filed some cases with Firebase support but they haven't provided any solution. Also, they are denying to provide ETA for all the support or feature requests. And with the launch of Firestore, it is highly doubtful that they will continue on enhancing and maintaining the Realtime database.
  42. It is not supported in China.
  43. Reasons Not To Use Firebase (a true story of Crisp chat system)
    1. Reasons Not To Use Firebase (Reasons Not To Use Firebase)
  44. Why I’m dumping Firebase for Web? - Michael Lugassy (Founder of Ad Tech is broken. We fix it!. Built 7 other companies, sold 3, worked with dozens more.)
  45. Why I’m dumping Firebase for Web? (Why I’m dumping Firebase for Web?)

So I would not say it’s bad but if you want firebase everywhere then welcome my friend, you are in hell.

Happy Coding,

Ridham Tarpara

Please don’t listen to the “proprietary blabla”. This is bs.

Imho MBaaS is the biggest failure ever made. No one ever will make money (long term) out of it. Their biggest failure, as you see in this post is, they don’t understand people like you. They think you will be a paying customer at some time. (This is never going to happen)

Now, let’s solve your problem. You’ve got that great idea and want to get going quickly. You seem like passionate about it. You want to focus on front end, not backend, means you focus and know your weaknesses. Very good point to start at!

So, why is Firebase your perfect solution? It’s supported by google. It’s super super easy to use. It’s well documented. Any more questions?

You can build whatever app (prototype) with Firebase in no time. It might not scale, but this is not what’s it about. You just want to master front-end stuff, don’t you? Right.

Eventually your MVP might get traction. What are you going to do? Well, if it’s going to get “a lot” of traction, you can pay someone to build a scalable backend for you. If not, scalability will never be an issue. You just don’t have to care about it!

Build whatever you like. Focus on the things you love to do. Firebase let’s you do exactly this. Keep it going, cheers.

ChitChak - Live Photo Sharing - Android Apps on Google Play

Check out our app here. This app is base on Firebase as backend entirely. From the experience of building this app, I can now quite confidently tell you that Firebase is not suitable to large scale project. An app like ChitChak that we build has hit some problem with limitation of Firebase. If I have to give one thing that bother me the most, it will be the problem when you wish to have some server side logic. Because you essentially do not have your own server side to begin with. So in order to do that, you can implement a server app (we are using node.js) that listen to the change of the real-time database and respond to the change. Or you can provide a few API and have you app consume the API to achieve something you can’t with just purely Firebase. However, this seems like defeating the purpose of using Firebase as a backend service.

My suggestion for large scale and complex app is that you may consider using Firebase for user authentication or some sub-features of your app. Especially those that need real-time update (eg. chat, comment, likes, vote & etc.). The real-time database is really good at that and can save you tons of time of building your own Push service. You may also treat the real-time database as a Message Queue like services and implement the Pub/Sub or RPC system around it, but do not expect it work as efficient as a real MQ broker. Some other feature like Ad-word, analytic, and deep-linking are also useful and easier to enable by using Firebase provided libraries. Since they are now belong to Google which is the provider for some of this services, I think the support of all this services will only get better.

Lastly, if you are building a prototype for some prove of concept, it definitely help you to complete you prototype in shorter time. It’s super easy to use with SDK and it’s currently totally free within a reasonable quota.

Yes but you will burn way too much money. Why?

Firebase is a general purpose very easy to use Api and so are the features. Very easy to use but not too rich of functionality. (you cannot get complex queries with Firebase e.g.)

You will easily build a Instagram Clone. However, at a certain point it’s simply too expensive and you’re better off developing your own stack to serve your needs efficiently.

Conclusion: Build an MVP with Firebase if you have no idea about backends. If your app gets traction you can easily transition to your own stack. (Most of the times you won’t get traction so there’s no need to waste time implementing your own stack)

The biggest reasons that I see are:

  • Cost. It’s pretty expensive when you start getting in the 1–10 TB+ range, compared to other cloud options or self-hosting.
  • Performance. Cloud databases or infrastructure can be a hindrance if you need peak performance.
  • Privacy Concerns. Some businesses are just worried about security and prefer to keep their data behind a VPN, sitting in a building where they know exactly where it’s at and who has access to it. I’m not saying this is actually safer, but many company owners think this way.

This applies to all cloud infrastructure/managed infrastructure too, not just Firebase specifically.

It looks like Firebase/Google has resolved the problem in an update, but as someone who is naturally paranoid, I second the motion of making sure that your back end infrastructure can be swapped out no matter the SaaS provider.

Yes, it's more work. But ultimately it's worth it.

If they hadn't gotten the attention of someone who could fix things, they would have been out a lot of money, or their service would have been killed. And of all of the positive things that I can say about Google, good customer support doesn't make the top 100.

I'm still planning on using Firebase for various products moving forward, but will continue to keep my servers in the loop.

An answer to this question can literally be found on the Firebase home page[1].

Footnotes