This Jez Humble talk about Continuous Delivery is one of my all-time favorites. As a bonus at the end he also totally demolishes the Google Guy.
Lovely to see the Twisted community take the lead in moving Python forward and I fully support this idea of giving the standard library the Konmari treatment.
Amazon RedShift is a hosted version of ParAccel. More recently, a plethora of open source SQL-on-Hadoop projects have emerged; they are young but aiming to compete with commercial data warehouse systems. These include Apache Hive, Spark SQL, Cloudera Impala, Facebook Presto, Apache Tajo, and Apache Drill [52, 53].
In these situations, as long as people agree on what the format is, it often doesn’t matter how pretty or efficient the format is. The difficulty of getting different organizations to agree on anything outweighs most other concerns.
Therefore, to maintain backward compatibility, every field you add after the initial deployment of the schema must be optional or have a default value.
That means you can only remove a field that is optional (a required field can never be removed), and you can never use the same tag number again (because you may still have data written somewhere that includes the old tag number, and that field must be ignored by new code).
If you are using a system with multi-leader replication, it is worth being aware of these issues, carefully reading the documentation, and thoroughly testing your database to ensure that it really does provide the guarantees you believe it to have.
Today, most data systems are not able to automatically compensate for such a highly skewed workload, so it’s the responsibility of the application to reduce the skew.
Unfortunately, these tools don’t directly translate to distributed systems, because a distributed system has no shared memory—only messages sent over an unreliable network.
Safety is often informally defined as nothing bad happens, and liveness as something good eventually happens.
A much better solution is to build a brand-new database inside the batch job and write it as files to the job’s output directory in the distributed filesystem, just like the search indexes in the last section. Those data files are then immutable once written, and can be loaded in bulk into servers that handle read-only queries. Various key-value stores support building database files in MapReduce jobs, including Voldemort , Terrapin , ElephantDB , and HBase bulk loading .
A complex system that works is invariably found to have evolved from a simple system that works. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. John Gall, Systemantics (1975)
When copies of the same data need to be maintained in several storage systems in order to satisfy different access patterns, you need to be very clear about the inputs and outputs: where is data written first, and which representations are derived from which sources? How do you get data into all the right places, in the right formats?
It would be very natural to extend this programming model to also allow a server to push state-change events into this client-side event pipeline. Thus, state changes could flow through an end-to-end write path: from the interaction on one device that triggers a state change, via event logs and through several derived data systems and stream processors, all the way to the user interface of a person observing the state on another device.
But this choice is not free either: if a service is so popular that it is “regarded by most people as essential for basic social participation” , then it is not reasonable to expect people to opt out of this service—using it is de facto mandatory.
A frankly amazing piece by Monica Lent using her experience to debunk seven commonly held truths about programming with verve. Programming consists of folklore and beliefs are often just beliefs. That indeed makes most learning in this field a question of unlearning.
A nicely put explanation for why I too pick my environments to be maximally productive with minimal configuration. In many cases, I just make do to the disgust of those who’ve learned all the shortcuts and applied all the customizations, but to them I’d say: marginal returns and opportunity cost.
I’d dare say that with the waning of the iPhone as a platform (both for development and for business), Apple’s closed plays will be increasingly lackluster. I’ve been rooting for Swift but it’s a horse running in the wrong race.
Gitlab’s CEO Shadow Program is a testament to their deep and deliberate thinking about organizational development. I’m curious whether more companies do this.
“code is just a high precision design medium”https://twitter.com/PavelASamsonov/status/1127207381673418752
From an interesting discussion of the nuances between product and design strategy a notable tidbit that sheds a lot of light.
Also while we are discussing the ever-shifting definition of design, here’s another interesting tweet.
You see this a lot with design wanting ‘a seat at the table’ but more often than not at those tables (which are inexplicably always at the highest echelons of power either in companies or government), most designers will not have the first clue what to contribute.
The issue around the Python standard library finally comes to a head thanks to a Twisted contributor. The standard library has felt stale for a very long time now. It’s nice that it’s there but nobody can really program anything in Python without external packages. There really is no debate here and core Python should support actual programmers.
“Changes to one team’s service may be implemented by another team who needs the enhanced capability by what is called an Away Team. This team works on the Home Team’s code to add what it needs according to established engineering standards and then leaves that code in good order to be maintained by the Home Team who owns the service, with help when needed.”
Fully embedding your team in another team’s context to get something done, seems to me a very interesting compromise between autonomy and collaboration. Every team is an island, but it is ok to travel to another island and if that’s not possible, you can still build your own.
Also more than disorienting to see an in-depth piece about engineering organizations like this appear on The Register.
Putting an entire team in another team’s context is something I have considered for all the collaboration, delivery, culture etc. benefits that it will most likely yield in exchange for a minor hit in delivery.