He doesn't understand the fundamentals of the problem he's solving, and is actively writing code that basically puts tools down and starts shouting "EVERYONE STOP!" in response to expected scenarios.
Most filesystems provide no guarantees at all by default on writes. NTFS journals metadata writes, but not data writes. Append-only files are absolutely expected to be truncated. The application must deal with this either by being insensitive to rollback, or by explicitly requesting a write cache flush! This is very well known to anyone that has ever written any kind of write-ahead-log, database engine, etc... There's a bazillion articles about how this is intended behaviour and no amount of screaming and shouting will make it go away. Learn the storage API guarantees, or STOP writing code that has critical dependencies on storage!
This quote especially seemed childish and immature to me:
> "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"
No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested, consumer HDD and SSD drives are well known to ignore this and keep things in volatile RAM cache to boost their IOPS numbers at the expense of durability. Only "enterprise" drives honour this API properly, and even then there are corner-cases around things like 512/4K sectors and torn writes. Similarly, consumer drives don't have power protection, so partial sector writes are entirely possible and should be expected if they lose power mid-write, or just crash at an inopportune time.
To be more constructive: The correct thing to do is that a log writer must always end log writes with a checksum of some sort. If the log is truncated (for any reason!), then it must recover starting from the last-known-good checksum. Throwing your hands up and saying "NO MORE BACKUPS FOR YOU! EVER!" is not the right response. Retrying backups from the last-known-good point automatically is the correct response.
PS: Some of his other rants are also a lack of understanding of thread safety and/or the lack of ECC RAM in consumer-grade PCs causing random misbehaviour. Heck, you'd also have to deal with bad sectors, corrupt filesystems, and a bunch of other corner cases that just makes this guy scream and shout instead of writing robust code...
PPS: I just realised that the reddit post is by 'CTO and Founder of Backblaze'. Wow. Note to self, do not use Blackbaze for anything, ever. If the CTO is this clueless, then their products can't be trusted.
it's even worse: there's no guarantee that if you don't touch something on the filesystem it will not be corrupted. It used to be almost the case with HDDs but you're using SSD and NVMe devices. Basically, it can happen that when there's a sudden power off, they will do weird stuff like "set the 6th bit of every byte in an entire region". In essence, whatever your strategy for making/keeping a summary of the source side of your backup, you will always need to be able to figure out the state/diff from scratch. This needs to be work. Everything else is merely there to speed things up.
Checksums are reasonably robust against this kind of thing, but there are storage systems out there (most SSDs!) that can even reorder blocks! So you can have valid checksums but still get corrupt data…
Disclaimer: I'm the engineer you are commenting on, I just want to straighten out one mis-understanding.
> This quote especially seemed childish and immature to me:
>
> > "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"
>
> No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested
Several times you seem to jump to the assumption that I don't understand fsync and disk flushing and that is the core issue. You aren't understanding what I'm criticizing. Here is an example of what bothers me:
You take a picture at your wedding, and you store it on your hard drive. You like the photo, it means a lot to you, and you use it as the background for your desktop FOR FIVE YEARS. You have rebooted hundreds of times, and it's always the background for your desktop. Then one day 5 years after your wedding, you reboot your laptop, and it seems to take a little longer to boot, and then after you sign into the laptop half the image you use for your desktop background is scrambled. The middle of it looks like dirty snow. You didn't get any reports of any issues from the OS manufacturer, but now one of your photos is corrupted.
This isn't because the software that wrote the photo 5 years earlier forgot to flush the picture to disk. It just isn't. Behind the scenes, as your laptop was booting from an aging drive, it probably encountered some issue, and it went about fixing the problem as best it could - which I have no problem with. My issue is the drive lost some data, and if the OS manufacturer would let you know this occurred you could take useful actions like order a new drive, prepare a restore from a few weeks ago before that issue occurred, etc.
> No, this is NOT Microsoft's or Apple's fault.
It isn't their fault that the drive is going bad, I agree. Drives go bad, that's why we have backups. My issue is the OS manufacturer try to cover up too much, keep too much hidden from the user, and didn't let the user know data loss has occurred (or might have occurred). And yes, I hold them accountable for not telling customers what is going on. It isn't anybody's "fault" that it occurred, but there is a responsibility to let customers know about it so the customer can take appropriate actions so they don't lose data (or more data).
I try to write incredibly paranoid software. Part of the reason is that is the "average" environment the Backblaze client runs in is more unstable than what most software developers are used to. The whole point of backups is to run when the computer is going sideways, it has bad RAM, it's losing disk sectors, or a customer's cat likes sleeping on the keyboard because it's warm, and the fans are clogged with cat fur. And because the family has teenage children that don't know about computer security problems, they download and install unstable junk from all over the internet because why not? That's the environment my software runs in, and I take my job of trying to protect my customer's data very seriously.
He doesn't understand the fundamentals of the problem he's solving, and is actively writing code that basically puts tools down and starts shouting "EVERYONE STOP!" in response to expected scenarios.
Most filesystems provide no guarantees at all by default on writes. NTFS journals metadata writes, but not data writes. Append-only files are absolutely expected to be truncated. The application must deal with this either by being insensitive to rollback, or by explicitly requesting a write cache flush! This is very well known to anyone that has ever written any kind of write-ahead-log, database engine, etc... There's a bazillion articles about how this is intended behaviour and no amount of screaming and shouting will make it go away. Learn the storage API guarantees, or STOP writing code that has critical dependencies on storage!
This quote especially seemed childish and immature to me:
> "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"
No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested, consumer HDD and SSD drives are well known to ignore this and keep things in volatile RAM cache to boost their IOPS numbers at the expense of durability. Only "enterprise" drives honour this API properly, and even then there are corner-cases around things like 512/4K sectors and torn writes. Similarly, consumer drives don't have power protection, so partial sector writes are entirely possible and should be expected if they lose power mid-write, or just crash at an inopportune time.
To be more constructive: The correct thing to do is that a log writer must always end log writes with a checksum of some sort. If the log is truncated (for any reason!), then it must recover starting from the last-known-good checksum. Throwing your hands up and saying "NO MORE BACKUPS FOR YOU! EVER!" is not the right response. Retrying backups from the last-known-good point automatically is the correct response.
PS: Some of his other rants are also a lack of understanding of thread safety and/or the lack of ECC RAM in consumer-grade PCs causing random misbehaviour. Heck, you'd also have to deal with bad sectors, corrupt filesystems, and a bunch of other corner cases that just makes this guy scream and shout instead of writing robust code...
PPS: I just realised that the reddit post is by 'CTO and Founder of Backblaze'. Wow. Note to self, do not use Blackbaze for anything, ever. If the CTO is this clueless, then their products can't be trusted.