Highlights for Rust for Rustaceans

For example, there cannot be two parallel flows with mutable access to a value. Nor can there be a flow that borrows a value while there is no flow that owns the value.
Freeing the memory twice could have catastrophic consequences.
If you just want to leave some valid value behind, std::mem::take 2 is a good candidate. It is equivalent to std::mem::replace(&mut value, Default::default()); it moves value out from behind the mutable reference but leaves a new, default value for the type in its place.
but as we dive deeper into the more complex parts of Rust, you will need a more rigorous mental model to work with.
The aim of this chapter has been to establish a solid, shared foundation that we can build on in the chapters to come.
False sharing occurs when two different CPUs access different values that happen to share a cache line; while they can theoretically operate in parallel, they both end up contending to update the same single entry in the cache.
Simply stated, the orphan rule says that you can implement a trait for a type only if the trait or the type is local to your crate.
For example, consider a type like SshConnection, which may or may not have been authenticated yet. You could add a generic type argument to SshConnection and then create two marker types: Unauthenticated and Authenticated. When the user first connects, they get SshConnection. In its impl block, you provide only a single method: connect. The connect method returns a SshConnection, and it’s only in that impl block that you provide the remaining methods for running commands and such.
you can see the building blocks in the RawWakerVTable type in the standard library.
In a way, unsafe is misleading as a keyword when it is used to allow unsafe operations through unsafe {}; it’s not that the contained code is unsafe, it’s that the code is allowed to perform otherwise unsafe operations because in this particular context, those operations are safe.
In practice, the safety and performance trade-off for unchecked methods is rarely worth it. As always with performance optimization, measure first, then optimize.
and then document them rigorously.
Not all code is written in Rust. It’s shocking, I know.
Instead, as shown in Listing 3-2, we can introduce a generic parameter on Rocket, Stage, and use it to restrict what methods are available when.
Rust Fuzz Book (https://rust-fuzz.github.io/book/)
Rust Cookbook (https://rust-lang-nursery.github.io/rust-cookbook/), which suggests idiomatic so
the Tokio project has published mini-redis (https://github.com/tokio-rs/mini-redis/), an incomplete but idiomatic implementation of a Redis client and server that’s extr
Philipp Oppermann’s Writing an OS in Rust (https://os.phil-opp.com/) goes through the whole operating system stack in great detail while teaching you good Rust patterns in the process. I also highly recommend Amos’s collection of articles (https://fasterthanli.me/tags/rust/) if you want a wide sampling of interesting deep dives written in a conversational styl

Highlights for It’s Not About the Burqa

I might not have heard the word ‘feminism’ yet, but I knew that the way women and girls were treated in Saudi Arabia was wrong and that this was not the Islam I was taught, nor did it represent the home I was raised in.
I am Indian. Yoga runs through my blood, it’s as natural to me as my vitamin D deficiency.
And yet I was surrounded by a community that disapproved of this sort of independence. Any attempt to challenge the traditional roles was met with disapproval that quickly spread through gossip and manifested as social control.
Ideally the hijab mitigates instances where a woman is valued solely on her appearance and sexuality – though whether it successfully does that in such hyper-sexualized societies is a whole different discussion – rather it aims to place worth on her intellect, her actions, her character, and so forth.
hijab was and is supposed to be an expression of faith and Muslim identity – that’s where it began, and that is where it was supposed to end.
We find ourselves trying to categorize our decision by placing it in a framework that negates the idea of Islam entirely – a framework that believes religion to be contingent, merely a set of historical practices and rituals, that believes in a complete separation of religion governing our affairs; the idea we stick to because our religion dictates our way of life.
But no, I am just the frumpy hijabi on the tube, supposedly beaten into covering myself in this sweltering heat, a mute with no voice and no brain, indoctrinated by an extremist ideology and with no opinion of my own.
It was only after that appointment, once I had a name for what I was going through, that I started researching Islamic ways to cope with my emotions. But when I looked online, on social media and Muslim forums, I was struck by the overwhelming prevalence of one single idea: that you could not be Muslim and depressed, because a true Muslim would be content with what God had planned for them.
Therefore, when a Muslim whose mental health issues are tied up with one of these turns to the community, they often find nothing but judgement, when what they seek is the relief promised by the Islamic principles of mercy and forgiveness.
Muslims are constantly being reminded that we are all one ummah – that we are of one body, and when any limb aches, the whole body reacts with sleeplessness and fever.
But when other Muslims see that you are not praying, regardless of the reason, they can be extremely judgemental.
But there is a problem. This perspective disapproves of the hijab, the burqa, modest culture and other key elements of the Muslim female identity.
Mainstream feminism suggests that my choices and values can’t exist within its framework – if I make the decision to dress for my faith then I must be oppressed or submissive.
I grew up in a Muslim community whose cultural understanding of Islam denied equality of the sexes and rarely left room for female voices, let alone female empowerment.
They can’t entirely explain or point to the oppressor.
I don’t know a single woman who will tell their male manager that it’s period cramps that are hampering their work, and women who need to borrow a paracetamol or a sanitary towel hardly shout it across the office.
I think my father thought that this sudden determination to wear a hijab was down to religious zeal. I can’t really remember. I only knew that wearing it meant that you belonged.
Hijab has served me well. At times, it has covered my scars, allowing me to wear long-sleeved tops without anyone questioning what was hidden underneath. Other times, it has served to cover my earphones while I avoided listening to teachers drone on in class. Sometimes, very rarely, it has kept my head warm during cold winters. My hijab gave me a way to act, a code of conduct: smile courteously at strangers, open doors for people, help the elderly carry their shopping, and politely decline drugs/alcohol/male interaction as they are ‘not allowed in Islam’. My hijab was my armour, something for me to fiddle with when people asked me uncomfortable questions. It would allow me to look down and cover the acne growing on my forehead when someone attractive walked by. At times when I was tired or frustrated, I would untie and retie my hijab. Now, I do so with my hair. It’s not the same.
The fact is, shame is one of the biggest drivers of toxic masculinity within South Asian culture and especially amongst Muslim men.
and until we stop mollycoddling Muslim men there won’t be any substantial change.
There is a difference between guilt and shame. Guilt comes from recognizing one’s own mistake. Shame is heaped upon us by others. And there is a place for shame in society. It should be heaped upon the patriarchal cultures that subjugate women. It should be felt by the women who allow it to continue, both through their silence and their actions. It should be placed upon the men who stand by and allow their mothers, their sisters, their wives and their aunts to oppress women in the name of Islam, men who benefit from their privilege. And it also belongs to the men who abandon us to its effects, simply because they are too afraid to speak up.
My husband told me later that his father had an aversion to skirts and saw my wearing one as a personal affront. He had an aversion to and an opinion on many things, it would turn out.
The actual granting of my divorce was as simple as that, because Islam makes it that easy. It was culture and its contradictions that made my life complicated.
After my second divorce my father told my mother: ‘You will never stop my daughters doing what they want again.’ Having raised us as equals with our brother, he had had enough of his girls being maligned. After this, we stopped pandering to the community. Outwardly, I merged my eastern and western wardrobes, mixing kurtas with jeans and shawls. Inwardly, I stopped giving a damn about gossip. The worst had happened. I was the talk of the town and there was nothing more to lose.
My mother, of course, was still concerned. What decent man would marry me, especially now I was on TV? Nice Pakistani girls did not appear on screen.
The mindset that a woman might do the unthinkable and refuse a man, or that she could even kill a man through the shameful use of her sexuality, means that men believe women need to be controlled, and this theory is conveniently backed up by supposed theology, providing a safety blanket for a wider community that is polluted by chauvinism and fragile male pride.
I wonder if I would have been protected from the heartbreak and pain that came as a result of trying to please a community that demanded I live by their rules only.
Because women in such unregistered marriages cannot get a legal divorce, if the husband refuses to give an Islamic divorce, they are referred to as ‘chained women’.
Imams told me repeatedly that they were under pressure from their congregation not to register marriages as it would lead to women having legal rights.
People are surprised to learn that it is harder nowadays to get out of a UK mobile phone contract than it is to leave an Islamic marriage.
When Muslim women choose to take a stand and vocalize our opinions, there are always consequences to our dissent – especially because it flips the orientalist caricature of a passive, repressed woman being held hostage by the men in her community.
Timothy Mitchell, in Colonising Egypt, explains how colonial officers drew ‘a link between the country’s “moral inferiority” and the status of its women’. He explains that they regularly came back to the argument that ‘the retarded development of the nation corresponded [ . . .] to the retarded development of the Egyptian woman’.

Highlights for Becoming Trader Joe

We witnessed something very interesting: the United States had a quota for imported tuna. Once Peru’s quota had been filled, a biological miracle occurred right there on the canning line. What had been tuna was now pilchard, a member of the herring family, on which there was no quota.
The Adamsons were land-rich but cash-poor; they lacked the funds to develop the magnificent property. They were having to sell off pieces for far less than they potentially were worth.
The convenience store business is 90 percent real estate, 10 percent all other (merchandising, personnel, etc.). In real estate, it’s the tenant’s balance sheet that counts.
The question was never on the employment application forms, and it’s probably verboten to ask these days. But dyslexia lurks in the brain of every left-hander, which means, we see the world differently, sometimes profitably. That’s why, when I interview people, I try to get them to write something. At one point I was accused of running a cabal of left-handers at Trader Joe’s.
Today it would be fought over by the vulture capitalists, but neither ventures nor vultures were common in those days.
If all the facts could be known, idiots could make the decisions. —Tex Thornton
Trying to find an optimum solution in business is a waste of time: the factors in the equation are changing all the time.
This is the most important single business decision I ever made: to pay people well.
We really didn’t pay more per hour than union scale, but we gave people hours. Because union scale is so high, the supermarkets are very stingy with hours and will do anything to avoid paying overtime.
Time and again I am asked why no one has successfully replicated Trader Joe’s. The answer is that no one has been willing to pay the wages and benefits, and thereby attract—and keep—the quality of people who work at Trader Joe’s.
Much as I would like to pose as an altruistic visionary, my policy was grounded partly by the desire to stay un-organized by the Retail Clerks Union, which under the direction of the legendary Joe DeSilva terrorized the market industry of those days.
The problem with unions is not their pay scales; it’s their work rules and seniority rules.
In fairness, however, those work rules did not spring from the ground like dung beetles working it over. They came from inexcusable employer practices.
The buyers at the supermarket chains knew nothing about what they sold, and they don’t want to know. What they did know all about was extorting slotting allowances, cooperative ad revenue, failure allowances, and back-haul concessions from the manufacturers.
But the Byzantine management atmosphere at first Rexall and then Hughes Aircraft had convinced me that the only real security lies in having your own business, and this left-hander was well ahead of the curve on that one.
Also, I was convinced that I was on a holy mission in preserving a company owned significantly by its employees. My hope was that someday it would be 100 percent owned by them. On that one I proved to be wrong.
That’s why, throughout my career, my policy has been full disclosure to employees about the true state of our affairs, almost to the point of imprudence. I took a cue from General Patton, who thought that the greatest danger was not that the enemy would learn his plans, but that his own troops would not.
It absolutely addressed our prime market, the overeducated and underpaid people of California.
Being king of the low-price, high-value wine trade in California was one of the greatest satisfactions of my career.
But bran is a low-value product. They couldn’t afford to deliver it. Since they also packaged nuts and dried fruits, however, we somewhat reluctantly added them to the order. And that’s how Trader Joe’s became the largest retailer of nuts and dried fruits in California!
In the Insider’s Wine Report we gave the results of the wine tastings that we were holding with increasing frequency, as we tried to gain product knowledge. This growing knowledge impressed me with how little we knew about food, so in 1969, we launched a parallel series of blind tastings of branded foods: mayonnaise, canned tuna, hot dogs, peanut butter, and so on.
It’s the problems that create the opportunities. If a business is easy, every simple bastard would enter it.
For forty years supermarkets in California had operated on a simple formula: run weekend ads, promoting Best Foods Mayonnaise and Folgers coffee below cost to get the people in the door, and sell them full-profit milk and alcohol.
The progress of the internet and electronic interfaces is demanding new levels of trust between the retailer and the supplier. Under these new interfaces, the supplier automatically resupplies the retailer on the basis of scanning data, which goes online to the supplier without batch-by-batch purchase orders. Individual internet buying must also operate on severe rules of trust.
In 1982, we employed an Apple II to do most of the number crunching. That was a big help, but it didn’t solve the problem of the nightly communication to the bakers. Young Joe rigged up one of the first voice-activated computer systems in the U.S. to take the orders from the stores. It was daring and full of bugs, but it began to teach us about electronic ordering.
created an electronic ordering system on the Macs that fed into Guy Lundberg’s computer service. This was an enormous breakthrough. You need to understand that we were outsourcing not just the mainframe number crunching but the printing of the documents that it generated. High-speed printers were a big choke point. They had to spew out “picking” documents for the warehouses, “receiving” documents for the stores, and summaries for our Accounting Department.
Look at any supermarket ad. You’ll learn precious little about the provenance of any product in it; you’ll see only name, size, and price. Partly this is because the grocers themselves don’t know anything about the provenance of what they sell, and they don’t want to bring up the subject of individual differences.
A distinction between full-time and part-time is a false dichotomy, when it comes to productivity.
The real limit on what range of products we could carry was our product knowledge. I believe that the greatest advantage of a limited-SKU retailer is that the employees at all levels can become truly knowledgeable about what they sell
Giving discounts to people over sixty is, to borrow a phrase from Charlie Munger, “a type of dementia I can’t even classify.” Here you have the fastest-growing, most affluent part of the population, and you give them a discount? If anyone should get a discount, it’s the shrinking workforce, which subsidizes the old folks through their income and social security taxes.
All members of Central Management, including myself, worked in the stores those days as a matter of morale for the troops.
The same mentality that doesn’t want to rock the boat also tends to hold onto its job. You’ll encounter those guys again and again! Try to help them do a good job; make them feel important; and make them feel that you’re Playing the Game, not trying to abolish it.
Believe me, you have a system for everything that has to happen in your business—you just may not be conscious of it. And you probably have still other systems that are not needed. That’s why The Winning Performance calls for a “constitutional contempt for business as usual.” To practice “constitutional contempt,” you have to arrive at work every day with the attitude, “Why do we do such-and-such that way? Better yet, why do we do it at all?”
Why Did I Want to Get Rid of This “Free” Labor? Some of those guys steal. They steal by doing “fast counts” with your receiving clerk (some of those route men should be dealing blackjack in Vegas). That’s one big reason why the supermarkets went to “electronic receiving.” They also steal by putting high-value items in the “empty” cases they take back to the trucks. That was one reason I welcomed L’Eggs in its egg-shaped containers in the late 1960s: it was harder to hide L’Eggs in an empty soft drink case than flat-pack hosiery. Ask any grocer about this. The bread and cupcake guys are under heavy penalties if they bring too much “stale” back to the bakery. So they pick up out-of-code product from one store and “roll” it into the next. As the drug culture developed in the wake of Vietnam, route men evolved into natural distributors of the stuff. You never know when they’re going to arrive at the store: this makes labor scheduling difficult. Related to this . . . They always show up in your parking lot when it’s jammed. Or have you never been blocked from a parking space by a beer truck?
Because of internal theft, one of the most important non-merchandise suppliers may be a detective agency. It is very hard to find effective agencies. The nature of the work tends to attract unstable people.
And with eleven years of Stanford education between us, we could never be poor, just temporarily impecunious at times.
Not until months after Dan arrived did Thrifty Drug begin to have even the most basic financial reports needed to operate a $1.8 billion business.
This is one of the most important things I can impart: in any troubled company the people at lower levels know what ought to be done in terms of day-to-day operations. If you just ask them, you can find answers.

Wat is geweld? vroeg hij haar. Is het besmeuren van gebouwen gewelddadiger dan het blijven verbranden van kolen, terwijl we weten dat de broeikasgassen die daardoor in de atmosfeer terechtkomen een hoop leed veroorzaken, niet alleen voor toekomstige generaties, maar ook voor de minder gefortuneerde mensen die vandaag leven?

Good article about Malm. The answer to the question above should be: “All climate action is self-defense.”

It’s possible to draw a straight line from Malm’s book to the climate protests happening right now in Berlin where people block the city ring road, which makes him quite effective.

This is why recent developments in fields such as Afrofuturism and Black horror are so crucial. They provide a critical alternative to the alt-right’s exterminationist fantasy of an all-white future. Just as importantly, they offer readers other ways of thinking about time that do not fall in line with the fascist dream of a history that unfolds step-by-step along the lines of the Aryan dictator’s master plan.


Year Note 2021

The days are long but the years are short.

One more year over and these are starting to look like a blur, so here’s what changed.


I gave notice between Christmas and New Year 2020 and spent the first three months of the year in a weird in-between state mostly done with the old job, waiting for the new one to start, in between daycare closures and being sick.

Then the new job started and I fell into a blur of work that has continued for most of the rest of the year. It’s been a nice change but also very intense. Details are on LinkedIn.


I haven’t been anywhere in 2021 except to an island in Croatia last September with work which definitely was a highlight.

Will I travel again and why? I have no idea.


I finished Turkish. I now prefer it over German again especially for writing.

I gave up on Chinese and couldn’t motivate myself or figure out how to book the HSK2. I did learn a couple hundred characters which seem to mean the same thing as in Japanese.

I started learning Japanese which is a lot more enjoyable and for basic conversation seems a lot easier. There also has been some strengthening on the claim for a relationship between Turkish and Japanese so let’s see how it goes.

I should have started this way earlier and not listened to anybody who ever said something is difficult. What they mean is that something was difficult for them. Depending on their intelligence, many things may in fact be difficult for them. It shouldn’t matter to me or anybody else.


The daycare was closed at the start of the year until March or Easter, not sure anymore. Most of the rest of the year things were up and running.

We had a nasty burn wound on one kid that cut the summer break short and gave us several weeks of grief but in the end all turned out well.

Then at the end of the year we gave notice. We are leaving one very esteemed Kindergarten for another which I didn’t think would be possible with two kids in the middle of the year, but it is. We even had several more places willing to take our kids.

The kids are doing really well. After a brief bout of gymnastics which they enjoyed a lot, COVID made us switch to outdoor ice skating for now.


I finally killed my old fixie or to be more correct it nearly killed me. With it gone, I could finally shop for a real road bike.

Thanks to a ship blocking the Suez Canal and all the other supply chain difficulties in the world, there was a tremendous bike shortage most of the summer (who knows, maybe it’s still going on?) and I got any random bike I could get my hands on in the €1000 price range.

I also got most of the tools and kit that you would need for cycling. It’s the clichéed guy midlife hobby. The local Rapha store is more convenient and also cheaper than many of the cycling apparel you can buy dropshipped over Instagram.

I seem to have ridden 1231km on it and looking forward to doing many times that in the coming year. The details are on Strava.

I picked up bouldering again which when in a rhythm (every 4-5 days) I’m finding pretty enjoyable.


I got boosted randomly on November 17th which was a lot earlier than I expected and before almost everybody I know. I am now AZ/Comirnaty/Comirnaty.

We survived another year. That’s it. That’s the year note.

Learn You a Haskell is Problematic

A couple of years ago I learned Haskell and to do that I read a book that is universally recommended for beginners and used to be available free online called “Learn you a Haskell for Great Good”.

The website seems to be gone at the moment: https://www.learnyouahaskell.com/

I didn’t think too much about it when I read it but after some other people were asking what they should use to learn Haskell I remembered these weird ‘jokes’ that are in the book to make the code a bit livelier.

Judging from the reviews online not that many people (in this case almost only men) have picked up on this. To be clear, it’s in extremely poor judgement to have picked these slurs in the first place and maybe you could get away with this stuff 5-10 years ago, that’s no longer the case.

I’ve screenshotted the examples below. There must be better books to learn Haskell with by now. I quickly switched to Clojure quickly for a bunch of reasons and you might do well not to take the detour that I did.

A really badly chosen and phrased BMI calculator
Not even clear what the point of this slur is
Is this a joke? Does the author have a girlfriend?

There could be more but I think you get the point.

There is a lot of truth to this Economist article about German humour and I refer back to it a lot.

Shortly after moving back to Germany in 2012 after decades of absence, mainly in Anglo-Saxon countries, I took my kids to the Berlin zoo. The children were two, four and seven at the time, and had already developed a keen sense of irony – or at least they understood that dad doesn’t always mean things literally, because, you know, it’s funny. So we queued for our tickets, trading silly jokes. Like me, the kids are dual citizens of America and Germany, though at that time, fresh from California, we still felt more American and more at ease in English. But we deliberately spoke German, to help us acclimatise to our new home. In a mood of levity, we approached the ticket window.

The lady behind it informed me that the price for the elder two was such-and-such and the littl’un was free. “What if I pay you a bit extra and you keep them?” I suggested. The kids chortled and started naming prices that might clear the market.

The lady stared back, horrified. Then, slowly, she leaned forward to look at my children, who stiffened. “Your dad does not really mean that,” she said. “He does not really want to sell you.”

That pretty much killed the mood for all four of us until somewhere between the giraffes and the polar bears. “Why did she say that?” my daughter asked, in English, as though out of an instinct for cultural self-preservation. As I pondered the question, I couldn’t help but think there was something peculiarly German about the lady’s reaction. First, Germans really, really struggle to grasp non-literal meanings. Second, Germans really, really can’t help but say when they think you’re wrong.


This opening story on literalism is great but there are more points in there, like the need to correct people and how isolating and grating this is for foreigners.

The point I take from it which also relates to the fact that street slang is segregated in Germany (as opposed to France). Educated Germans will not use words from the street register in normal conversation for fear of looking uneducated. You can see this for instance in the criticism Rezo gets for his use of mixed language in his serious videos. ‘Wrong’ use of language is used as an excuse to put people down. This happens to me as well every time I make a mistake.

The counterpoint is that anything said in High German is considered to be true or at least intended to be a statement of fact. The literalism is the baseline for all interactions at that level. They are literally incapable of feigning.

You can imagine the problems this creates and how unfixable they are because they are so deeply embedded in the culture. The only way out is through. The Rezo generation needs to create a new culture and the old needs to die off.

Leveled up in Turkish

Hanging out on Clubhouse from the beginning made me realize how lacking my Turkish was. I got on the app when Germany came online and I think Turkey launched shortly after that and only then did it kick off in the Netherlands.

That particular sequencing gave me an interesting perspective on how different cultures adopted the service. Germans went into panel hell. Most of the sessions were dominated by journalists and politicos and therefore were extremely boring. Turkey got online in no time and was dominated by influencers who launched all kinds of funky repeating formats and casual chatrooms where people were going until deep into the night (I would listen until 1-2 at night which in Turkey would be 2 hours later still). The Netherlands went into a marketing fervor with a strong showing of douchebags.

Clubhouse had its moment and fizzled, but tuning into the app again recently, I can say that it’s still there and with all the klout and hype chasers gone, things are again weird. Weird is good.


In the Turkish rooms I would hear words that I didn’t know every other sentence, sometimes entire concepts like kadının beyanı esastır. Finally, I had a situation where I was exposed to a high level of Turkish and I had the motivation to improve this.

I started noting these words down in Anki, which I had bought previously to study the HSK. Adding a word to my deck here and there was slow: switch to dictionary (now I use the official one), switch to Anki and write the card, switch back.

For some extra speed, I dug up a master word list I had made more than a decade ago. Back then I had read most of Orhan Pamuk’s then bibliography (Yeni Hayat, Sessiz Ev, Kara Kitap, Kar, Benim Adım Kırmızı) and then too found lots of words I didn’t know. I passed over them and wrote them down on a piece of paper. I transcribed these papers into a list and looked up some of the meanings. But this being before Anki, I had no real way to systematically learn these words and get better at Turkish. If I had, I would have improved steadily with each book I read and everything would have been great.

I went through this old word list letter by letter and over the course of a month made cards for all of the words in that list. The result of that effort is the 1000-word deck I have made available online now.

And now, after months of revising daily (life during the pandemic has been very exciting, why do you ask?) and also continuing to add words as I go—the recent events in Afghanistan made me add the half-Arab half-Farsi ridiculous Turkish word for charge d’affaires—I finally am out of “New” words. From now on it will just be polishing the lexicography of the deck, adding a word here and there and continuing to revise.

This also means I have capacity again to go back to studying HSK3. Or I might switch back to learning Japanese after all. Watching some anime recently and seeing Japanese speaking gaijin on Tiktok have revived this itch.

A feeling for Turkish (and Arabic)

So now what? I am a lot more confident when it comes to my Turkish after having learned a bunch of the words commonly in use in the higher echelons of society. I can now look for and find complicated words much more easily. My usage will likely be incorrect often, but I should be able to get by with my sizeable intuition for what remains the first language I learned (and learnd to read in). I can also consume complex material far more easily than I used to.

I now realize how many loan words from Arabic Turkey has. The exact numbers are obscure, but almost every uncommon word either is of Arabic origin or has an Arabic equivalent. This also explains why after being in Syria for a while I understood what people were saying without knowing Arabic.

I’ve also been amazed by how overloaded the language is for certain concepts like disaster (afet, badire, facia, buhran), sorrow (ıstırap, üzüntü, matem, yas, kahır, hicran, gam, tasa, keder, nedamet, teessür) and many others. Also all of those words are Arabic in origin except if I had to guess by word shape: üzüntü (I guessed right!).

White Turks

Delving into this part of the language and the people who use it, I came across the concept of White Turks. The division between white and black Turks underlies a lot of the dynamics of the Turkey of the past decades.

The people who I was listening to and whose language I am now emulating are usually white Turks. Me and my family originally are black Turks.

I’m sure I’ll never pass as white, though I’m now in a socio-economically better position than most white Turks in Turkey and most black Turks in Europe. Language is a key aspect of this division and when people clash it is usually the first weapon that they resort to. Now that I have levelled up, I don’t have to be exclusively on the receiving end of that weapon anymore.

Highlights for Site Reliability Engineering

Traditional operations teams and their counterparts in product development thus often end up in conflict, most visibly over how quickly software can be released to production. At their core, the development teams want to launch new features and see them adopted by users. At their core, the ops teams want to make sure the service doesn’t break while they are holding the pager. Because most outages are caused by some kind of change—a new configuration, a new feature launch, or a new type of user traffic—the two teams’ goals are fundamentally in tension.
SRE is what happens when you ask a software engineer to design an operations team.
The use of an error budget resolves the structural conflict of incentives between development and SRE. SRE’s goal is no longer “zero outages”; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. This change makes all the difference. An outage is no longer a “bad” thing—it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.
When humans are necessary, we have found that thinking through and recording the best practices ahead of time in a “playbook” produces roughly a 3x improvement in MTTR as compared to the strategy of “winging it.” The hero jack-of-all-trades on-call engineer does work, but the practiced on-call engineer armed with a playbook works much better.
However, some systems should be instrumented with client-side collection, because not measuring behavior at the client can miss a range of problems that affect users but don’t affect server-side metrics.
Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.
At least 50% of each SRE’s time should be spent on engineering project work that will either reduce future toil or add service features. Feature development typically focuses on improving reliability, performance, or utilization, which often reduces toil as a second-order effect.
A product’s feature velocity will slow if the SRE team is too busy with manual work and firefighting to roll out new features promptly.
This kind of tension is common within a team, and often reflects an underlying mistrust of the team’s self-discipline: while some team members want to implement a “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it’s easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes. Managers and technical leaders play a key role in implementing true, long-term fixes by supporting and prioritizing potentially time-consuming long-term fixes even when the initial “pain” of paging subsides.
It’s easy to overlook the fact that once you have encapsulated some task in automation, anyone can execute the task. Therefore, the time savings apply across anyone who would plausibly use the automation. Decoupling operator from operation is very powerful.
The main upshot of this new automation was that we had a lot more free time to spend on improving other parts of the infrastructure. Such improvements had a cascading effect: the more time we saved, the more time we were able to spend on optimizing and automating other tedious work.
“Why don’t we gate the code with a flag instead of deleting it?”
If we release 100 unrelated changes to a system at the same time and performance gets worse, understanding which changes impacted performance, and how they did so, will take considerable effort or additional instrumentation. If the release is performed in smaller batches, we can move faster with more confidence because each code change can be understood in isolation in the larger system.
There are many ways to simplify and speed troubleshooting. Perhaps the most fundamental are: Building observability—with both white-box metrics and structured logs—into each component from the ground up. Designing systems with well-understood and observable interfaces between components.
Some on-call engineers simultaneously experienced what they believed to be a failure of the corporate network and relocated to dedicated secure rooms (panic rooms) with backup access to the production environment.
Google relies upon our own tools. Much of the software stack that we use for troubleshooting and communicating lies behind jobs that were crash-looping. Had this outage lasted any longer, debugging would have been severely hindered.
De facto, the commander holds all positions that they have not delegated.
It is important to define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary. In addition to these objective triggers, any stakeholder may request a postmortem for an event.
Writing a postmortem also involves formal review and publication. In practice, teams share the first postmortem draft internally and solicit a group of senior engineers to assess the draft for completeness. Review criteria might include: Was key incident data collected for posterity? Are the impact assessments complete? Was the root cause sufficiently deep? Is the action plan appropriate and are resulting bug fixes at appropriate priority? Did we share the outcome with relevant stakeholders?
Make sure that writing effective postmortems is a rewarded and celebrated practice, both publicly through the social methods mentioned earlier, and through individual and team performance management.
one of SRE’s guiding principles is that “team size should not scale directly with service growth.”
Performance Data describes how a service scales: for every unit of demand X in cluster Y, how many units of dependency Z are used? This scaling data may be derived in a number of ways depending on the maturity of the service in question. Some services are load tested, while others infer their scaling based upon past performance.
When deploying approximation to help speed development, it’s important to undertake the work in a way that allows the team to make future enhancements and revisit approximation.
By working one-on-one with early users, you can address those fears personally, and demonstrate that rather than owning the toil of performing a tedious task manually, the team instead owns the configurations, processes, and ultimate results of their technical work.
Load test components until they break. As load increases, a component typically handles requests successfully until it reaches a point at which it can’t handle more requests.
If you believe your system has proper protections against being overloaded, consider performing failure tests in a small slice of production to find the point at which the components in your system fail under real traffic
Its authors point out [Bur06] that providing consensus primitives as a service rather than as libraries that engineers build into their applications frees application maintainers of having to deploy their systems in a way compatible with a highly available consensus service (running the right number of replicas, dealing with group membership, dealing with performance, etc.).
Regardless of the source of the “thundering herd” problem, nothing is harder on cluster infrastructure and the SREs responsible for a cluster’s various services than a buggy 10,000 worker pipeline job.
We don’t make teams “practice” their backups, instead: Teams define service level objectives (SLOs) for data availability in a variety of failure modes. A team practices and demonstrates their ability to meet those SLOs.
Google has also found that the most devastating acute data deletion cases are caused by application developers unfamiliar with existing code but working on deletion-related code, especially batch processing pipelines
The most important principle in this layer is that backups don’t matter; what matters is recovery.
Was the ability to formulate such an estimate luck? No—our success was the fruit of planning, adherence to best practices, hard work, and cooperation, and we were glad to see our investment in each of these elements pay off as well as it did.
In short, we always knew that adherence to best practices is important, and it was good to see that maxim proven true.
At first, this race condition may occur for a tiny fraction of data. But as the volume of data increases, a larger and larger fraction of the data is at risk for triggering a race condition. Such a scenario is probabilistic—the pipeline works correctly for the vast majority of data and for most of the time. When such race conditions occur in a data deletion pipeline, the wrong data can be deleted nondeterministically.
The Google Search SRE team structures this learning through a document called the “on-call learning checklist.”
When standard operating procedures break down, they’ll need to be able to improvise fully.
Because of the rapid change of production systems, it is important that your team welcome any chance to refamiliarize themselves with a system, including by learning from the newest, rather than oldest, members of the team.
At some point, if you can’t get the attention you need to fix the root cause of the problems causing interrupts, perhaps the component you’re supporting isn’t that important.
Once embedded in a team, the SRE focuses on improving the team’s practices instead of simply helping the team empty the ticket queue. The SRE observes the team’s daily routine and makes recommendations to improve their practices.
A default to ops mode usually happens in response to an overwhelming pressure, real or imagined.
Any serving-critical component for which the existing SREs respond to questions by saying, “We don’t know anything about that; the devs own it” To give acceptable on-call support for a component, you should at least know the consequences when it breaks and the urgency needed to fix problems.
Usually, the SRE team establishes and maintains a PRR checklist explicitly for the Analysis phase.
For example, SRE might help implement a “dark launch” setup, in which part of the traffic from existing users is sent to the new service in addition to being sent to the live production service. The responses from the new service are “dark” since they are thrown away and not actually shown to users.
What happened The effectiveness of the response What we would do differently next time What actions will be taken to make sure a particular incident doesn’t happen again