Might not help your deploy process if webhooks are degrading, but at least it can keep your remote in sync with local changes, if you need to deploy from local.
My wife and I have been playing this game every day for the past few weeks or so. Some of the daily levels are incredibly satisfying.
We have very different ways of approaching the harder problems; some of the clues are easier for someone with a formal CS/engineering background, but others just require a better memory of the different constraints, and she's much better at building a tagging system around those constraints.
Hatchet | Product Engineer | NYC or SF or REMOTE (US and EU) | https://hatchet.run
Hey HN! I'm Alexander, one of the founders of Hatchet. Hatchet is an open-source platform for running background jobs at scale.
We're hiring product engineers who are excited to build the next class of engineering primitives, starting with queues, background tasks and durable execution. We started in early 2024 after launching our distributed task queue (https://news.ycombinator.com/item?id=39643136).
Hatchet is currently used by thousands of engineers for all kinds of workloads: log ingestion pipelines, code review agents, video encoding, GPU scheduling, etc. Our target customer is fast-growing startups who have a strong need for background jobs system. These days, that tends to be AI companies, though we're general-purpose and not exclusively targeted for AI workloads.
Applying: I'm trying something different this time. `ssh hatchet-jobs.com`
Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)
I just started a new role and were using hatchet at scale. So far I have found it to be very well thought out.
I love creating webhooks to test an event then deleting the webhook afterwards, makes testing for the QA team easy.
> I do wish the guide was expanded a bit with lots of specific examples and details... a lot of it is hand-wavy blurbs.
I appreciate the feedback; I'm usually someone who tends to go into way too much detail, so this was difficult to write - I tried to focus on the "mental model" of understanding Postgres rather than very nuanced specifics. I tried to link out to my favorite articles on a number of subjects, and the Postgres manual is quite good.
> in addition to minimizing locked records, make sure your locks are ordered deterministically across all queries (eg by id asc, always) or you’ll deadlock (but postgres has a really good deadlock detector so you’ll more likely just error out if you’re lucky)
This is really good advice, I should put this somewhere in the guide. To add to this, not only can you deadlock by not having a consistent `ORDER BY` when you're locking sets of rows, but you should also be careful of locking rows on tables in different orders. For example, even if you lock each row in a table with an ORDER BY and FOR UPDATE, if one tx locks `table_a` and then `table_b`, and the other locks `table_b` and then `table_a`, you'll deadlock. This is obvious in theory but exponentially harder to debug in practice, because you need to be globally aware of every table that a write touches - something that's bitten us in particular with certain extensions.
> learn about GIN (and GIST) indexes
We're just testing GIN for fast key-value lookups for JSONB columns, and the performance improvements have been really massive. Interestingly there was a large performance skew between AND vs OR on these key-value queries.
OP here, I appreciate the feedback! I tried to focus on things which could take down your database, so things like particularly slow reads and writes, autovacuum settings, reducing lock contention, particularly focused on cases that I've seen. There are lots of things that I left out which would belong in a general user guide.
We're heavy users of stored functions because we're (perhaps overly) reliant on Postgres triggers, which can improve performance by reducing network round-trips but are fairly risky because they're difficult to monitor and observe.
Hatchet | Full-Stack Engineer | NYC or SF or REMOTE (US and EU) | https://hatchet.run
Hey HN! I'm Alexander, one of the founders of Hatchet. Hatchet is an open-source platform for running background jobs at scale.
We're hiring engineers who are excited to build the next class of engineering primitives, starting with queues, background tasks and durable execution. We started in early 2024 after launching our distributed task queue (https://news.ycombinator.com/item?id=39643136).
Hatchet is currently used by thousands of engineers for all kinds of workloads: log ingestion pipelines, code review agents, video encoding, GPU scheduling, etc. Our target customer is fast-growing startups who have a strong need for background jobs system. These days, that tends to be AI companies, though we're general-purpose and not exclusively targeted for AI workloads.
Applying: email me at alexander@hatchet.run and tell me about something impressive you've built, along with your CV and why you're interested in Hatchet.
Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)
> Our analysis revealed the fork-to-star ratio as the strongest simple heuristic for identifying potential manipulation. The logic is straightforward: a star costs nothing and conveys no commitment. A fork means someone downloaded the code to use or modify it.
This is just clearly...incorrect? You can both modify code without forking it and most software is distributed via a registry or binary download, which also wouldn't be represented in forks. For most projects, the number of forks is a lossy signal for how busy the contributor ecosystem is, nothing else.
Candidly we're still trying to figure that out: all of the plumbing is there in the open source, but the actual implementation of writes to S3 are only on the cloud version. This is partially because we're loath to introduce additional dependencies, and partially because this job requires a decent amount of CPU and memory and would have to run separate from the Hatchet engine, which adds complexity to self-hosted setups. That said, we're aware of multi-TB self-hosted instances, and this would be really useful for them - so it's important that we can get this into the open source.
The payloads are time-partitioned (in either case) so we do drop them after the user-defined retention period.
Going to create a PR into https://github.com/knobiknows/all-the-bufo later today.