This is the problem with most of the guys who go with MongoDB. Obviously, this person is very technical, so I am not flaming him nor accusing him, but this is my view of the rest of them who pick MongoDB without exactly having a clue as to why (hipsters) or when they should use a NoSQL db and when they shouldn't.
I do not hesitate to admit that I was a hipster sometime back too. I chose MongoDB for many of my projects and it went well, till it reached some kind of moderate scale where I realized it was a terrible choice going with a NoSQL db (Sometimes, I'd have to duplicate data because there were no Joins, etc). And that's when you start to realize, that NoSQL is not a pure-white solution. It is designed to satisfy very specific use-cases. Relational databases are really good enough for 99% of the use cases out there.
Unless otherwise you are COMPLETELY unable to design your schema in a relational database, you SHOULD NOT simply opt for a NoSQL database. The claimed NoSQL performance benefits will easily be outrun by a terribly designed schema, if you use the wrong db for the wrong scenario. Trust me, MySQL has had so much negativity because of these hipsters, but even something as basically relational as MySQL scales really reaaallllly well. Infact, many top guys still use MySQL till date, for a reason, in production.[1]
Next time you launch your start-up, spend some time carefully evaluating your db design decisions, as the wrong db for the wrong use-case could easily become the most expensive mistake of your startup.
Disclaimer, I work for RethinkDB a competitor of MongoDB.
I completely agree with you that NoSQL is not a pure-white solution but I disagree that it's only useful for specific use-cases. NoSQL has frequently been touted as this magic concept that makes databases scale well. It doesn't as many of the poorly scaling NoSQL databases on the market today show you. The most you can say is that it maybe lets you ignore a few particularly hard to scale operations. However there's also no reason that a NoSQL database can't scale well it's just that you have to actually design it in such a way that it does and Mongo failed at this.
Joins are a very useful feature of structured databases but there's no reason you can't have them in a NoSQL database. I speak from experience here, RethinkDB has joins (distributed joins no less). They're just as useful in a NoSQL world as they are in a SQL world.
I think the author's analogy "NoSQL : SQL :: Dynamic Typing : Static Typing" (to use SAT syntax) is a good one. I know some damn good programmers who prefer dynamically typed languages like LISP and I know some damn good ones who prefer statically typed languages like Haskell and I think programmers can be similarly productive with either a NoSQL or a SQL databases backing them up. Currently NoSQL has the very real downside that none of the NoSQL databases on the market have really nailed it (ours included but we're working on it) whereas there are some very well established and well tested SQL databases on the market so it's a perfectly rational choice to say you prefer the better tested SQL route. However I contend there's nothing inherently flawed about the NoSQL concept.
It’s a distributed near–real‐time querying system for data at rest that supports subset of SQL.
Google has scalable SQL database called Spanner[1]. They also have a proper NoSQL database called internally Megastore[2] that provides distributed transactions based on entity‐groups on top of BigTable. Also known as High Replication Datastore and available via AppEngine[3].
All I can say is this: if the saying "Always plan to throw away your MVP" is true, then I can't see any other storage solution other than MongoDB (or a similar schema-less document storage DB) for MVPs. The speed of development and flexibility are simply worth it. Yes, it is hard to refactor a live product and move it from MongoDB to MySQL / Postgre but was done before and you only do that if you get traction, so its a good problem to have.
I would start with MongoDB, get a grip on what on earth the product is doing, finalize the schema on the fly based on A/B tests, customer feedback and analytics, only once the schema is finalized move it to a SQL database if needed.
If you are not using a good ORM + DB migration system, then MongoDB will make perfect sense when you are quickly iterating through ideas and trying to find a product / market match. You really have no clue how your data schema is going to look like in the end, why confine it at the start? the vast majority of startups sadly won't get even near the phase where they really reach scale related performance issues, so choosing MongoDB for prototyping your business absolutely makes sense to me.
You make a VERY big "if" in your first sentence, one that (admittedly anecdotally) I've very rarely seen hold true in tech companies. Much more often, the MVP becomes the product, and all those shortcuts and poor design decisions come back to kill your productivity when it becomes necessary to refactor foundational tech/designs that have metastasized throughout the codebase.
I am curious if this is other folks' experience as well, or do you actually throw out the MVP and start all over at some point? If so, at what point do you make the break?
In real life, you are totally true. The initial test that succeeds becomes/continues to be the real product. I agree with the parent poster that Mongo is awesome for testing, if not awesome for scale. I think the point here is that tech co-founders/leads need to make it clear that this is a debt that will need to be paid if things take off.
If NoSql's speed and flex gets you to MVP, and MVP gets you traction, then all the hell you may go through to rearchitect for SQL is probably worth it.
Premature scalability is as dumb as premature optimization and premature generality.
When I develop against MongoDB in python, ruby, or clojure, I consider it less a database and more a persistence layer for my dicts/hashes/hash-maps, with a handy-dandy query functionality built in.
Hell, in clojure I usually develop against a hash-map of hash-maps stored in-memory until the project gets far enough to re-implement the data layer (which I've hopefully abstracted well enough to not be a big deal).
Think of Mongo less like a schemaless postgres, and more like a persistent redis. Ease-of-use is its true advantage, and I think anyone who exceeds its capacity should be happy to have the problem.
I tend to treat Mongo similarly. At my job, we hate Mongo and I've stood up and cursed it more than once.
Plenty of personal projects end up running on Mongo, because it has a rich query layer for quick hacking. I also never expect these to exceed the amount of RAM on my machine either.
This is really the killer feature of MongoDB its API embeds very nicely in host languages and is very intuitive to use. I work for a competitor called RethinkDB and we've spent a lot of effort trying to make something that can compete in terms of ease of use. The leap we're really trying to make is for people to not have to throw their database out with their MVP.
I've worked for 15 years with relational databases and for the recent 6 months with mongodb on my startup. For me, the agility of not having to define my data in advance, and "work" so much to explain the DBMS how my schema should behave is no less than a game changer in my ability to build the MVP quickly.
I just can't imagine having to invest all the labor in having to model everything I did with a normalized database and ORM.
Really?
Mongo is for me not a database. And I worked like you 15 years with RDBMS and a year with mongo. Mongo has no security. Especially the Mongo Cluster Solution makes it so easy to steal and manipulate datas without notice.
Everyone who has access to the server can enter the database. Implemented "security" could be turned off with a restart.
And nowadays it is so easy in a datamodel in a RDBMS to make changes. It is high flexible too.
But another worst thing in Mongo is the performance at complex queries.
It is a nice bin for thinks nobody needs or to put documents in but that's it with Mongo.
I don't understand how this argument keeps coming up. If you use rails, schema changes and migrations are DEAD EASY and I am not even a professional developer. What time exactly are you saving ??
Please don't conflate MongoDB and NoSQL. NoSQL is about the right tool for the job. For instance many of our Couchbase customers are very technical and risk averse. They choose Couchbase because other solutions (often MySQL or Oracle) have become the wrong tool for the job, as requirements change.
Mongo is a little different bc they actively position themselves as a MySQL replacement for your average Rails app.
* The job almost certainly will change. Data lives a long time, usually much longer than the original application.
* It's likely to conflate marketing claims with actual fitness for a purpose. A special-purpose system with special-purpose marketing may sound great if the special case lines up with what you're doing. But that doesn't mean that it's actually a better fit for that special purpose. And, thanks to confirmation bias, people almost always think their situation does sound like the special purpose for which the system was built.
* A database system is a system, not a tool. A tool is used in a particular instance for a short period of time, the results are obvious, and it's easy to switch tools if one is not working out.
Of course, there are always valid reasons to choose the special-purpose system, and I'm not saying you shouldn't have used Couchbase. I just don't like the "right tool for the job" argument.
I've heard DBA complain that many programmers see data as not all that important: that the data simply exists to run the program. This obviously not the case in any serious real-life environments, but I think it's the sort of attitude that many NoSQL supporters have.
One thing that become much easier with the JSON document model is sync, which is a valuable abstraction over the network layer. http://blog.couchbase.com/why-mobile-sync
When installing a database system, it is rarely a "task". It's usually something that will live a long time and there will be many long-term consequences.
I'm not saying that you shouldn't make short-term compromises. I'm saying that "right tool for the job" is sloppy reasoning.
Even in the short term, it might be much faster to develop against a good RDBMS than a nosql system. I think many developers are so focused on "abstracting away the data store" that they forget what a good RDBMS can do for them.
I would have to agree. Stackoverflow is in the alexa top 100, with over 400 million page views per month, and it runs off a single MS SQL Server instance (with a hot replica), with a Redis cache in front of it.
All the high-scale users of nosql systems end up having to learn a lot of arcane technical wizardry to make it scale (as demonstrated by the slidedeck linked here), and I'm not convinced it's actually easier to scale something like mongo than a traditional RDBMS.
No , there is something specific to MongoDB. Some NoSQL dbs are fine for some purposes ( multi-master sync for couchdb ,graphs for Neo4j ... )
MongoDB problem is that it promises too much and fail to deliver on a lot of levels. You cant expect to do all the operations one can do on a RDBMS on a NoSQL database and still scale horizontally,while maintaining data integrity. A good NoSQL db should have little features , but features that work very well. I like CouchDB for instance i use it in my apps to store customer infos that need to be synced accross multiple device, but i would NOT build an ecommerce app with it only...
NoSQL dbs solves specific problems. It does not solve the problems relational databases are for.
So selling MongoDB as a replacement for PostGreSQL or Mysql is a blatant lie.It will not work.
I suspect the reason that Mongo attracts so much hate is that it's more heavily marketed/evangelised than other NoSQL DBs - so more in the public eye and people expect more from it. Since the reality is that Mongo is not yet a technically strong product (regardless of its merits in terms of ease of use and getting simple things done quickly), it's bound to attract more negative attention.
Bingo. My brief experience with Mongo a few years ago was in a situation where it got selected in order to use something "new and cool". My job was to produce some reports from the data. Guess what reports require? Joins. The whole mess was slow and sucked, and the client belatedly realized that MongoDB was a shitty choice for what they needed, which could have been handled by Postgres without breaking a sweat.
Having spent the last two years trying to make MongoDB work at large scale and only succeeding because I have a ton of resources (200+ db hosts), I can say it's more than just the spotlight.
Fair enough - I have no direct personal experience with Mongo myself. Coming from a more traditional DB background, what I've read indicates it has a lot of technological flaws, but I didn't want to be too harsh in my judgement without personal experience.
For my own curiosity, what issues caused you the most trouble? The lack of transactions combined with the extremely coarse grained write locking were what put me off the most.
Read performance compared to a lot of other offerings is a significant issue.
However, one of the biggest issue is the entire sharding design that is incredibly delicate and wouldn't pass even the most basic high availability requirements. I could go into a lot of detail about why it's bad but it would take too long.
After that, it's the hardcoded limitations that prevent true multi-datacentre sharding, the previously mentioned CPU issues with mongos, broken replica selection, inability to control primary/replica setting manually, etc.
I used to spend a lot of time in the RDBMS space, and I'd hear the DBAs bitch about how NoSQL was just a bullshit excuse for not having any discipline around your data model, and I'd defend it, saying, "Well, you have to understand, certain use cases, etc. etc. etc."
And then I see an presentation like this and think, "Yes, of course, you should have put that stuff in a relational database, what were you thinking".
Of course, most of the open source databases do sharding and replication and the like equally terribly, so it's not a perfect solution....
I tend to take the view of RDBMS works until it doesn't. Most cases that things like click data or event data that ends up taking billions of rows, in which case you aren't going to want it in a RDBMS anyways (you'd like to, but hah).
"till it reached some kind of moderate scale where I realized it was a terrible choice going with a NoSQL db (Sometimes, I'd have to duplicate data because there were no Joins, etc)"
That statement doesn't make sense to me. One of the first things you do with a sql database when you reach moderate scale is denormalize data so you don't have to do joins, or shard which prevents a lot of joins. Joins are super convenient, but they don't scale.
On nearly all of the cases, they are very usefull, but once in a while one of them gets too slow, so you make a bit more of work enforcing the consistence by hand, accepts a bit more of risk, and denormalize it.
Throwing all of them away because a few may create problems later is a bit of an over reaction.
How many joins do you think this page uses? A denormalized version that doesn't scale probably uses 25-50. I'll bet the scalable implementation uses close to zero.
I would argue that its actually a multi-tiered question where the choice isn't really "NoSQL or RDBMS" but "NoSQL or ORM talking to RDBMS or going straight to RDBMS"
Here, the naive developer uses MongoDB as an all-in-one that rolls in the part of ORM that too often gets sole consideration, 'How do I persist my objects?', and effectively ignoring everything else. Try as I might to eloquently phrase my frustration with this all, it usually ends up in a rant against those relying on automagic who don't feel the need to understand every decision being made for you.
I have to completely agree... in many cases a classic SQL RDBMS is probably best. In some (such as single records having very dynamically shaped data, where most instances can be brought forward based on a key alone, and searches are on a few keys), NoSQL is better. In some, having data replicated from one to the other works well, one for primary records, one for search/display. I've even contemplated scenarios where ElasticSearch could be a source record. It really just depends on what you want/need from your data and searching.
I do not hesitate to admit that I was a hipster sometime back too. I chose MongoDB for many of my projects and it went well, till it reached some kind of moderate scale where I realized it was a terrible choice going with a NoSQL db (Sometimes, I'd have to duplicate data because there were no Joins, etc). And that's when you start to realize, that NoSQL is not a pure-white solution. It is designed to satisfy very specific use-cases. Relational databases are really good enough for 99% of the use cases out there.
Unless otherwise you are COMPLETELY unable to design your schema in a relational database, you SHOULD NOT simply opt for a NoSQL database. The claimed NoSQL performance benefits will easily be outrun by a terribly designed schema, if you use the wrong db for the wrong scenario. Trust me, MySQL has had so much negativity because of these hipsters, but even something as basically relational as MySQL scales really reaaallllly well. Infact, many top guys still use MySQL till date, for a reason, in production.[1]
Next time you launch your start-up, spend some time carefully evaluating your db design decisions, as the wrong db for the wrong use-case could easily become the most expensive mistake of your startup.
[1]http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use...