> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write.
Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.
I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler.
> basically just one for system and majority of serving workload ran on another priority and the rest was for batch
RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Yes, just absolutely crazy way of doing load shedding.
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario.
> Application backends?
I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.
are these separate services? Are you going to integrate how services preempt into overall system design?
My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
> you want to move these decisions into the application tier as much as possible
I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.
Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).
This is an excellent book and its lessons saved my bacon many times!
As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.
I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)
actually, I think this time it is the other way around, Azure is more the solution than the problem.
Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.
Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.
Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent.
I remember running IIS and Apache side-by-side many years ago, and oh boy, was IIS a pain.
Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...
The article doesn't mention any Azure outages, but a client had an Azure hosted container restart that day which caused problems due to configs being stored out of persistant storage.
If you load an issue page, you'll see 1 failed request to:
/project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:
/in-product-messaging/copilot-budget-request-banner
/in-product-messaging/code-scanning-ai-findings-preview-banner
/github-copilot/chat
/_private/browser/stats
If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks.
It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error.
But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
Shyeah, all those successful requests were really helpful for showing me a page, which then failed when I tried to do anything by clicking the buttons on the page. I was completely unable to merge things for hours, for no lack of trying. GH was down.
Thank you so much for this gem, this sent me through the rabbit hole and I'm fascinated by the level of engineering in this book. I love how big complex systems are engineered. It feels like anything is possible when you have a solid plan and you keep improving it step by step. So inspiring.
Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle.
Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.
Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.
Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.
I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.
And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.
You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.
You don't have a single Google datacenter (or any other provider / big name) for a single tenant. The bigger the DC, the more the number of tenants/customers reaches "infinity" (giving the same meaning to "infinity" you are giving to it)
Sure there is infinite capacity given an infinite amount of time to scale up. I assert you’re leaving out the time dimension. Those so-called infinities are simply not accessible in a practical way since you’ll hit a wall in actually provisioning that capacity long before the data center runs out of compute.
> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity
This.
It's easy to be an "armchair quarterback," here, though.
Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.
+1. I also felt this huge gap between knowing (intellectually) what parenting is and actually experiencing it. It really changes you, to the point that now, when I talk to childless people my age, I feel like we're fundamentally differently. I think I can perceive fairly accurately whether someone my age is a parent after just a few minutes of general conversation (obviously, without having to visit adjacent topics).
I experienced this so strongly I spent some time actually writing a poem (in Spanish) about it: https://alejo.ch/39l
> It really changes you, to the point that now, when I talk to childless people my age, I feel like we're fundamentally differently.
As a childless person, the inverse is equally true. I have wildly different levels of obligation and constraint - and that difference typically becomes obvious very very quickly. I don't say that as a better/worse, just different.
Interesting perspective. Your first paragraph also describes my experience, perfectly.
However, even though I also think nothing comes even close to parenthood for me, my perspective is that society strongly encourages —I'd even say strongly pressures— people to have kids, to feel that if you don't, it's almost like something is wrong with you. Maybe we're just different generations? :-)
I never felt that pressure to have kids. In fact, even though I waited until I was in my mid 30's, I was the first in my social circle to have kids (I guess I hung around with younger people?).
I knew since I was a kid myself that I wanted to someday be a father. If society pushed me, it would have been when I was still in elementary school.
After I had kids though… It was strange to scroll through coworkers social media feeds and see the same photos of groups of 30-somethings holding up wine glasses while the waiter (I assume) takes their photo at some fancy restaurant in the Bay Area. It all seemed so… hedonistic? Or maybe just kind of empty.
Becoming a father gave me purpose that was (finally) not about me any longer.
And now, the nest empty, I still am unable to go back to that person I was before. (Whoever that was.)
I live in a neighborhood of older retired folks and I get this vibe from them as well when walking my younger kids. Their faces light up with joy. Just another season in life I suppose. Let's enjoy it while we can.
Yeah, speaking as a female-bodied person here - undoubtedly a minority on this site - I absolutely feel the pressure to have children in society, pretty much every day.
Look at the "birth rate" discourse that's been going on. There's tremendous pressure on women to have more babies. There's also a lot of pendulum-swinging in the opposite direction with your edgy "antinatalists", in some sort of weird "overcorrection" or some such, but ultimately those voices are much louder online than in real life.
Fewer people are reproducing, but that doesn't mean the pressures to do so aren't still there. "It's complicated."
This onslaught of birth rate articles laying the blame at women's feet drives me up a wall. Many of my female peers have frozen their eggs to preserve their ability to have children to the best of their abilities or have undergone fertility evaluations proactively, other female friends in their 30s/40s have pursued the single-mom-by-choice route once they had the financial security to do so. I do not see my male peers pursuing sperm testing / freezing or pursuing single-fatherhood-by-choice. Of my married friends with kids, if there is a second kid it has been because the wife advocated for it. All that said, I don't blame men either the husbands that were talked into the second kid were all concerned about their ability to pay for school / daycare and/or to afford another bedroom even with the help of their working wives / breadwinner wives. For men talked into their first kid, their hesitation was that after undergraduate and masters they felt they'd barely had time to financially establish themselves or live their own lives.
Not to mention the evidence coming out about advanced paternal age and adverse outcomes on offspring. The pressures are definitely lopsided. If anything, the pressures should be on society and institutions to provide more support for parents, people who want to become parents, and people who just want to be part of "the village". So... support for people, I suppose. A rising tide lifts all boats. Something like that. I dunno. There's a lot of factors at play here; I'm just spitballing.
Does the birth rate discourse put pressure on you as an individual? I had expected that to also be more of an online than real life push.
There is more pressure on women to have kids than on men. That is not new. What is new is a more negative attitude to children and parenting in society in general. Do you feel there are conflicting pressures now?
>Does the birth rate discourse put pressure on you as an individual?
Implicitly, I think, yes. It builds on all of the other social pressures, big and small. I'm sure an entire humanities thesis could be written about how it all adds up over time.
I wrote another comment elsewhere in the thread talking about polarized opinions broadcasted on social media and how that also adds to the stress of it all. Nobody ever wants to talk about the "it was/is... fine, actually. kind of whatever. not great, but not horrible." possibility of pregnancy (and parenthood). It's always either the most wonderful, miraculous thing ever and you're insane for not wanting it, or it's the most horrible, irreversibly damaging thing ever and you're insane for wanting it.
And the thing is... it COULD be any of those outcomes! It could ruin your life. It could also make your life so much more fulfilling. There's simply no way to know for sure what it will be like for you without doing it. It is an enormous gamble, especially when you are the person who has to endure the pregnancy, and it is a gamble that you can't really walk away from. (Of course, you can give a child up for adoption, but there's a lot of social shame involved in that, especially if it is a wanted child that you're quite literally giving up on rearing.)
I think the other comment was in answer to me too.
I think you are right about the polarised opinions. I think in real life (at least for people I know) pregnancy is generally fine, with maybe a few horrible bits (i.e. if there are any medical problems/something goes wrong). It has lots of positives but its difficult.
Parenthood is usually wonderful. I am afraid men have it easier as we get the parenthood without the pregnancy!
It also depends on your social and professional background. If you are an orthodox jew, you will likely have the maximum pressure.
On the other hand, my wife, who was a lawyer, was fired from from a Big4 when they suspected that she was pregnant.
As a father in the tech/startup industry, it's sometimes quite hard to get people to understand what having children implies for instance, my ex-boss couldn't understand that I wasn't available at 21:30 everyday to answer his questions, as I was bedding the kids.
I mean, humans are made to comply to social pressure, so if they stop collectively having children, it more likely means that the social and economic pressure is overall rather against having them.
I think it's social pressure in some cultures, but even in those (sub)cultures which don't have the explicit pressure to have kids, people have a drive to procreate. I cannot tell you how many couples I know who decided they would not have kids, or would only have one, and at a certain point one partner (usually a woman in their late 30s or early 40s if I am being honest) decided this was the most important thing and no amount of money or career setbacks or relationship turmoil wouldn stand in the way.
Different generations, different places, exactly where you are in your society, religious influences, your immediate social circle, how much time you spend on social media....
Certainly in the UK there is a group of people who have become very negative about having kids, partly because they are scared of the expense, and partly because society makes it look so hard, and partly because they genuinely have less support.
Pregnancy is also a really scary prospect for a lot of people (myself included), and there doesn't seem to be much work done in trying to make it, well, less shitty of an experience. "It's just how it is," but... is it really? Does it really need to be so bad?
Not to mention that "it takes a village", but in many places the village has been basically stolen from us.
Pregnancy is never going to be easy (and its one part of parenting I am glad I did not have to do!) but it can definitely be made easier and more pleasant. it varies a lot though. We were quite lucky in many ways.
However, why is pregnancy scarier than it used to be when it is safer than it has ever been? It seems to be more scary in developed countries than in poor ones too.
I wonder whether the loss of the "village" might be part of the reason for this. it certainly is for a lot of other things around having kids.
I wonder how much of it has to do with the fact that birth control offers us a choice in the matter. Pregnancy used to be something You Just Did, because, well, most people enjoy having sex, and pregnancy is the intended consequence of unprotected sex. It's scary, it's stressful, it's risky, and it's sometimes even fatal... but it's "part of life". It's how we ended up here, after all.
Now we have a choice. Which is GREAT, amazing, even. Absolutely necessary. I'll never argue that anyone's right to choose if they want to be a parent is a bad thing.
I wonder if social media is also contributing, with its polarizing nature. It's rare you see any opinions or even anecdotes about pregnancy being "fine" or "whatever". It's always a big deal, and it's always a big deal (horrible) or a big deal (wonderful). There's not a lot of in between. The negative end of the spectrum scares people, while the positive end of the spectrum is met with skepticism since it's too-often (for me at least) correlated with right-wing religious people trying to push women to be what're effectively baby factories, so the "it's the most wonderful experience ever!" just comes off as disingenuous and manipulative.
Combined with the fact that women's healthcare is rife with challenges, and women's pain is often dismissed (not looking to start any "gender wars" arguments; just stating things that can be backed up https://pmc.ncbi.nlm.nih.gov/articles/PMC10937548/), the prospect of becoming pregnant is a scary one, when you have the option to not do that.
"What if it's nine months of constant nausea, being bedbound, gaining weight I'll never be able to lose? My body will change, in many ways irreversibly. Will my partner still find me attractive? Will they support me during? Will they support me after? Will I feel supported? Will I feel resentment that they'll be able to enjoy things like alcohol while I'll have to stay completely sober for a very long time? Will I feel resentment if the majority of childcare falls on me, which is statistically actually quite likely? What about my career? Will I regret it? What if my child is born with a disability? What if I'm bad at being a mom? What if I hate it? What if I'm trapped in a life I don't want? What if there's no way out?"
Of course, the above thought process could also be one with thoughts like "what if it's wonderful?" but it's a lot easier to worry about the negative, and put weight on the negative, than it is to fantasize and put weight on the positive what-ifs.
This is just my perspective, though. Some people love being pregnant and find parenthood really wonderful (or so they tell me about those things), and I love that for them. Truly. I'm just not sure if I have it in me to ever find out what it would be like for me, and I know I'm not alone. Fun Auntie is a better role for me I think. But maybe I would feel differently if pregnancy were easier and a less destructive process on the female body, and there was more social support for parents.
Birth control has been around for a long time now. The oral contraceptive pill was invented 70 years ago. Fear of pregnancy to the level of putting people off having kids is much newer.
I think its very likely that social media plays apart.
I think people glossing pregnancy over as wonderful are optimistic. I think a more honest take is "its hard but worth it".
> What if it's nine months of constant nausea, being bedbound, gaining weight I'll never be able to lose?
Its not nine months of nausea or being bedbound though. Most women I know lived normal lives for most of their pregnancies.
> Will my partner still find me attractive? Will they support me during? Will they support me after? Will I feel supported
Absolutely do not have kids with a partner whose support you cannot rely on!
None of us are going to keep our looks all our lives anyway.
> Will I feel resentment that they'll be able to enjoy things like alcohol while I'll have to stay completely sober for a very long time?
When i had my kids the NHS advice was to drink in moderation. Changes since then have been politically driven (and by increased American influence on attitudes) rather than being the rest of any new science. My ex went off alcohol before she knew she was pregnant (which is the most critical time) and then drank moderately thereafter.
> Will I feel resentment if the majority of childcare falls on me, which is statistically actually quite likely?
That is understandable. Again it partly comes down to having the right partner. Some men do end up doing the majority of childcare - I am one, and I have come across a few others. This is something that is improving too (I hope!).
> What about my career? Will I regret it? What if my child is born with a disability?
Careers get into perspective when you have kids. Very few people regret having their kids, whatever goes wrong. They are so much more important than having a career. I have a lot less money than I would have had without kids, but there is no way I would want to have the money instead.
> What if I'm bad at being a mom? What if I hate it? What if I'm trapped in a life I don't want?
IMO it is extremely rare for the sort of person who worries about being a good parent to be a bad parent. The problem is that people want to be perfect parents, and society encourages anxiety instead of reassuring them that it is OK not to be perfect.
> Fun Auntie is a better role for me I think
Its not a bad role to have! I would point out that being a fun auntie requires someone to have kids!
> and there was more social support for parents.
I think this is critical. I am not sure to what extent pregnancy can be made easier in a direct physical sense, but many women could be better supported through it.
All valid responses/pushback to the anxiety! It's still a lot of unknowns and what-ifs; I was more trying to make the point that for some people, it's overwhelming, and some may conclude it's not worth the risk in finding out. Which brings me to another important point, which is that parenthood isn't for everybody. Some people really shouldn't be parents, and there's nothing wrong with acknowledging that. (Not saying I'm necessarily one of those people; just speaking more generally.)
Unfortunately - and this is kinda related to your comment about it being rare for the kind of person who worries about being a good parent to be a really bad parent - we often see opposite patterns play out... people that would probably make good parents seem less likely to reproduce, and vice versa. I know "Idiocracy" is a bit of a meme these days and is arguably classist, but I can't help but think about it. It's probably a far larger socioeconomic issue than can be feasibly hashed out in an internet forum, and I certainly can't solve it.
Ultimately, I think where I'm at these days, is that we need the village back, and I would love to be a part of it. How? Well, currently, it's that "Fun Auntie" role, as I've no kids of my own, nor do I plan to change that. I've long "wanted to want" bio-children, but so far, I don't. I'm mid-30s now, so there is still time, especially thanks to fertility technology continuing to improve. I'm probably not gonna change my mind, but I'm far too autistic to speak in absolutes, so I cant 100% rule anything out.
Regardless: the village is a beautiful thing. It's an honor to be a part of it in the ways that I am and have been throughout my life. More people should be a part of it. We should all (within reason) be a part of it. It should be celebrated and supported each and every day, to the point where parenthood isn't so much of a burden on the individual family unit, and is instead a shared responsibility of the local community. How do we best go about doing that? Obviously governmental and institutional support is critical, but we also need societal/cultural support, and that will be a lengthier project... we owe it to ourselves to try, though.
I appreciate the conversation and insights in this conversation; thanks for the well-thought out responses in these threads.
I really appreciate how well you have explained a perspective very different from my own. Not only am I a man who has had children, my younger daughter turned 18 today, and will be off to university in less than two months so my children are no longer children and worries about parenting etc. are behind me.
Its interesting how much we agree about the social and political problems and the loss of the village.
Thank you for taking the time to type that, I found it informative and interesting. Like we like to think of it as a single idea that worked well, but it's actually a ton of work to get a million details just right, and failure in just a few of them would just have made the whole endeavor (of the printing press) fail.
there is also another question. the figure you cited for Germany is gross wage. net wage would be €3,324/month, or $45,536/year. is "income", in the US survey, after taxes? i wasn't able to find the definition of "income" in the American Community Survey, but i notice that besides "income", they also have "earnings" (which is higher).
finally, we need to correct for cost of living. i think that would explain a lot of the difference.
I went back to a senior IC role after managing for 5 years and I think there are big differences. Yeah, some aspects have to do with leadership (regardless of whether or not you manage) and there's overlap, but... there are also huge differences.
I wrote about one I experienced: https://alejo.ch/3g9
The approach we're taking to deal with this very real context rot is using a bunch of related techniques which we call transposing the agent loop: https://alejo.ch/3jt
In essence, we run many short agent loops, generating their prompts dynamically from structured data. Each loop advances the state in a small step towards the final goal.
Oops, you're right, I put the wrong url (Android auto complete fail, oops, suggested a link and I assumed it was the right one). I meant to link to https://alejo.ch/3gk
I guess I'll hide this submission and do another one with the right url.
But filtered coffee is the most flavorful! No other method extracts the subtle nuances as well!
P.s. I know it's subjective, just cringe on this claim of "the most flavorful" starting with espresso. :)
Yep, every method brings out a different nuance of coffee flavor, and any true coffee snob will likely own half a dozen or more items for brewing coffee.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.