I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.
I'm increasingly of the opinion that "basically a data warehouse, but incrementally maintained materialized views under the hood as an implementation detail" (e.g. Materialize, Feldera, anything based on Differential Dataflow / DBSP) is a good point in the design space. I get that they're more complicated than batch, and for many orgs the low latency isn't necessary for pure analytics/dashboards.
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
I’m not sure you have to squint; an mview is just a cached query. The only difference between an mview and a cache is the “incremental” part of the equation.
That is, my only disagreement is the lack of gusto
the fact that traditional database infrastructure handles many people's needs doesn't really obviate dataflow style architectures. personally I think (3) composition is a real potential win. I would also note that streaming is a really great base on which to build a distributed database that looks more like Postgres on top.
I guess I'm mostly confused about the idea that streaming systems should be trying to supplant sql databases, or that a failure to do so implies that they don't have utility. it looks like the GP has done some really interesting work on automatic parallelism. that isn't pointless just because most people building operational systems still reflexively reach for PG.
That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs.
Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans.
One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.
A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database.
It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies.
I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model.
Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true.
Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe.
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink.
Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.
1. completeness is still a very desirable behavior for many use cases, but it's gonna be tricky to build these use cases on top of the new wave of IVM-like streaming systems, which provides no completeness marker.
2. periodic queries declared in non-streaming declarative form can and should be computed incrementally, but as far as I know, there's still not much mature support for automatic optimizations
ABC was originally designed for European folk music [0], not church music. The corpora as a result is largely fiddle tunes, particularly Irish (see for example https://thesession.org).
ABC started very simple, because most of the performance information for folk music isn't written down, it's inferred by the player according to the idiom of that particular tradition. As usage of ABC has grown it's gotten more powerful but still falls far short of formats designed for western classical, like MusicXML or Lilypond.
MIPS is increasingly common in rock helmets. Black Diamond (anecdotally the most common brand I see in the US) now has it in their higher end models, and Mammut as well.
Just checked my BD climbing helmet I bought few years ago (seems like Mips Capitan model based on design).
Considered these things a gimmick (any technical equipment bought has list of various tech used within, I generally ignore that by default since I have no idea what each means), happy to see move for more safety in this area. Even small steps matter.
I will climb in big heat wave we have here in Europe now this evening, more sweaty = more slippery on polished rock crags, risk is always not as far as we like to think.
Petzl makes better helmets than BD. They pioneered the ultralight foam kind with the sirocco that everyone (including BD and Mammut) copied. Yet they still don't care about MIPS.
You do not need perfect pitch to quickly learn fiddle tunes by ear (source: I play fiddle, can quickly learn by ear, do not have perfect pitch). You learn tunes primarily by relative pitch, which most people can develop.
There's also an element of violin-specific pitch detection; if you play violin for long enough, you can recognize the specific timbre of different notes on a violin (particularly easy for open strings) which helps ground you while listening to a tune.
Actually shift should just partner with Airbnb. Airbnb takes out the cleaning fee from the app, customers benefit, owners get a free cleaning service, Airbnb wins, Shift wins. And if Airbnb invests in Shift, even better.
Airbnb should invest in Shift, continue to encourage exorbitant cleaning fees, and subsidize discounts for hosts so that they're incentivized to fire their current housekeeping providers and switch to Shift.
The fact that they're not partnering with Airbnb tells us what we need to know. The top of this thread is more likely now: they're interested in surveilling the people, not the buildings.
Most airbnbs have completely uninspired interior decoration but I had a great one that I'll always remember for having really unique lighting (lots of layers and textures)
The article specifically mentions at least one property owner who has been denied any recourse because of the lack of before/after photos (presumably before that specific rental).
I’m sure Airbnb operators get comfortable turning it over every few days without having to constantly take photos. Most guests don’t bring robots in to smash up the dishwasher and dent the walls
After thinking about this for a while, I'm not sure it really happened. It wouldn't surprise me if the house was not trashed, just a landlord manipulating evidence when they think they can make money in court. There is no particular reason to trust either side and we have not seen what evidence really exists. In particular the reporters didn't do a good job of digging in - at the very least where is the response from the Bot Company?
A company operating above board would be sure to carefully document the state of the rental before and after whatever work they were doing. Any tradesperson/installer/technician/repair person will have tales of how they were accused of stealing grandmas wedding ring from the bottom of the sock drawer while repairing a leak in the kitchen.
So either Bot Company damaged property and is trying to pretend they didn't. Or they are incompetent and failed to document the state of the property or handle the owners complaints appropriately.
Given that their training robots and would therefore be collecting as much data as possible, including camera data, I'm leaning towards malice instead of ignorance.
I don't entirely doubt the landlord but the bizzarre part is the landlord showing up to take their trash and then somehow finding bundles of wires inside the unit. Why would an airbnb host enter the unit to take trash?
That’s part of the process of resetting a property for the next reservation. It’s not bizarre, it’s literally what Airbnb landlords do (or sometimes hire other people to do, but that lowers margins)
"Why would an airbnb host enter the unit to take trash?"
Not every airbnb host has a professional cleaning staff, and some of those who do may sometimes wish to check the status of their property. I don't find anything strange, let alone "bizarre".
Does the un-regulation cut both ways? A landlord usually needs to notify tenants 24hr in advance if they're going to enter the property. Does an AirBnB host need to follow any similar rules? It's not like the renters have a lease, it's not their residence.. Do they have any rights to privacy or notice at all?
I don’t know why they are on there, but YC startups list their batch and year in parentheses on job posts, e.g. (W25). Example: https://www.workatastartup.com/jobs/88812
The Plaid listing you linked doesn’t have a batch by their name.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)