comments 6

All About Eve

Some Background:

Eve is a new programming language.  Maybe.  Or, a development environment.  More accurately, perhaps, it’s a tool to think.  If you have an hour to spare, this is a great video to watch from Chris Granger:

If you don’t then I’ll summarise (badly).  The video starts with the question “What will programming look like in 10 years?”.  Chris started off his career with Microsoft working on Visual Studio, but then, faced with the complexity of this product, wanted to build an IDE which stripped back a lot of the complexity, and Light Table was born (for editing Clojure / Clojurescript).  Light Table aims to minimise or eliminate the cycle between writing code and seeing results.

The lesson from this was apparently, that just fixing the IDE wasn’t enough.  Programming itself suffers from the problem that it is indirect and invisible, beleaguered with accidental complexity and unobservable.  

Paraphrasing heavily (really, watch the video), Eve started as a collection of experiments into how computing could be made better.    They extensively reviewed the existing literature and state-of-the-art, deciding that the closest we have come to really solving those issues of programming are in tools like Excel and SQL (i.e. relational databases).

Along the way, they throw out GUI editors, query visualisations and more traditional text based languages.  Chris then demonstrates version 0 of Eve, which is kind of a Wiki, combined with a query language, and kind of a database.  Everything updates instantly on screen.  There’s huge promise, and I am tempted to get involved in a big, open-source way.

Lotus Notes

My first thought on watching the demo was This seems a lot like Lotus Notes.  And indeed, Wikis and Notes share a lot of common thinking:  unstructured data, functions for manipulating values, views on data etc.

I can imagine that not many people working in IT today are familiar with Notes, but take it from me: this is a complement.    Notes was an amazing (and amazingly productive) environment to work in.   You could realistically build multi-user applications with workflow, email, reporting, everything.  The security was top-notch.  Lotus Notes came with an Email application, which was built in Lotus Notes, and during the 90’s millions of people used Lotus Notes for Email:  the point being, you could build amazing things in it.

So resurrecting a lot of the ideas of Lotus Notes now makes a lot of sense to me.  We don’t have anything similar to this now, and that’s a shame.  There’s a middle ground (somewhere between building simulations in Excel, and building full applications in Java) which is served badly since we don’t have Notes anymore.

So, good on them.  This seems like a really cool product with a bright future.


I Don’t Think This Is What Programming Will Look Like In 10 Years

If Lotus Notes was so cool, why aren’t we using it today?  Also, if programming in Excel is so awesome, why am I not developing banking software using it?

The problem is, (I think) tradeoffs.  And there are three I want to talk about:

1.  Openness and Accidental Complexity

When you have a closed system (like Eve, or WordPress, or Lotus Notes, or MySQL, or Excel) you can really put a lid on Accidental Complexity.  Accidental Complexity is things like:  in system A, I have a column called “Name”, but in system B, I have “First Name” and “Last Name”.  Or, here I have dates formatted in the American way, but the file I am importing has them in the European way.  Or, I have an event-driven system which triggers every time there is a new order, but my finance system requires a daily feed… those kind of things.

When you program in a general purpose language like Java, or C++ or whatever, you can deal with accidental complexity, because these languages are like glue: they can join together disparate libraries, or databases, or applications, or feeds, or queues or whatever.  They are “Open”:  they accept that the world at large doesn’t play to any particular rules, and that they need to adapt to the world.

“Closed” systems, on the other hand, force the world to play to their rules.  So, when I write a plugin for Eclipse, Excel, Minecraft, WordPress or whatever, I have to obey the rules.  That’s fine.  But, a lot of systems that we use every day are out there, not playing to the rules.  Certainly, there are no pre-existing systems playing to Eve’s rules, because Eve’s only just been thought of.

But, can this problem be solved by building “plugins” for Eve?  So that we can get it to talk to existing relational databases, or filesystems, or whatever.  Well yes and no.   Lotus Notes had this challenge already, back in the day.  People wanted it to talk with SQL databases.  Work was done so that data could be synchronised, and maybe that solved a lot of people’s problems.  But inevitably, with this, you imported the accidental complexity from those systems:  table structures, date formats, edit cycles, transactions etc. 

So, it feels to me that this is a fundamental issue:  can we build open systems without importing accidental complexity?  Because, if we can’t, then a lot of programming in the future can’t be done in an “Eve” way.

But, maybe that’s ok.

2.  Organisation vs Approachability

In Chris’ video, he talks about how in an early version of Eve, users were bedevilled by scope.  They wanted to use data in functions, but were constrained by not having the data in scope.  He observed that neither Excel or relational databases had this issue, and so, the Wiki-style Eve takes a leaf out of that book and jettisons the concept of scope.

But, scope is a useful thing that was deliberately added to programming languages to reduce errors.  Global state is rightly vilified as being a source of errors in software:  as soon as you scale your software endeavour, you run the risk of confusing pieces of global state.

If you look at many of the features of Java (say):

  • Separation Of Concerns
  • Packages / Namespace / Scoping / Jars
  • Interfaces
  • Abstract Classes / Type Hierarchy
  • Versioning
  • Methods / Procedures / Functional Decomposition
  • Compilation / Compile-time checking

These are all features to improve the organisation of the software, reduce duplication and enforce consistency through once-and-only-once.   Yes, you can build a much more approachable, easy to learn, less indirect language with none of these, but as the scope of your software problem gets larger, the bigger issue is organisation, not approachability.  So, we trade this off.

This is one of the big failings of Lotus Notes:  versioning the software was hard.  Upgrading databases was hard.  Knowing what code was even in the application was hard.  Testing was all manual.  Re-use was minimal.  Revision control (a la git) was impossible.   When you changed something in one place, you had no idea whether this had introduced a problem somewhere else.

And now, it’s actually worse than this.  Right now, I feel that Java (et al) fail us:  It’s still really hard to do releases, and test that software is correct, and ensure that changing a part of the system I’m building is going to have no consequences on other parts of the system.

My client has a vast network of interrelated computer systems, on different release cycles, on different hardware, with different programming languages.. how do we know this all works together?

Yes, pure functional programming would help.  Yes, better testing would help.  Maybe Docker, or Chef could help.

But, right now it’s a mess, and in 10 years time it will be an even bigger mess.   Java allows us to organise at the scale of one application.  But we need tools to help us organise our software at a scale of many applications.  Because, software is eating the world, right?

3.  Focus vs Coverage

Lotus Notes came along when the Internet was in it’s formative years.  So, when you started as a Notes developer, you got given a printed manual or two.  This was great:  everything you needed to know was in the manual.  And, you could be massively productive and run rings around the guys building C++ applications with RDBMS back-ends.  Cool.

What eventually went wrong (I think) was that, as web pages took off, people wanted their Notes databases on the web, and Lotus/IBM then bolted on loads of stuff to make Notes databases web-browseable.  But, it was hard to use and fiddly, and required learning HTML, and the whole conceptual integrity of the Lotus Notes model was lost (see issue 1 again).

The problem is:  one size doesn’t fit everyone.  Also, you can’t hope to keep up with everything.  Software systems at best can hope to serve one niche of people.

It’s the same reason why we don’t just build applications solely in SQL Databases:  at some point in time, companies like Oracle tried to build products where you could build the entire application inside the database, and it would have forms, and web pages and everything.  But, this didn’t work out either:  even ignoring vendor lock-in, it’s better to be able to mix-and-match your technology choices to fit to the problem you have:  trying to be all things to all people is just a non-starter.

This is why Lotus Notes couldn’t keep up: it was too busy trying to do everything, rather than one thing well.


And, maybe this is unfair.  What Chris and the Eve team have done is looking amazing.  I don’t think Chris really meant for Eve to be “what programming looked like in 10 years”.  Really, what he was saying  is “we need to start doing more experiments about what programming is, and what it should be, like we did for Eve.”

Eve takes a stance on the three trade-offs above, and it’s pretty much the same stance Notes took:  I loved Notes, and I’d like to see a modern version.   Preferably one which solves things like versioning and so on.

But, the design of “programming in 10 years” needs to acknowledge these trade-offs, and then design beyond them:  truly innovative design should reframe trade-offs in ways which completely change the landscape.  We’re only going to move beyond the current status quo with some amazing design, more experiments, more ideas and throwing away what doesn’t work.

Chris’ Comments

Chris Granger took the time to discuss some of these points with me, on Github:

Update #2

This was discussed further on Hacker News here.

comment 0

Homo Connectus

Just as Homo Sapiens (humans in tribes, hunting and gathering) evolved into Homo Economicus (humans as  a whole ecosystem, using money to decide who exploits which niche), we have evolved again.

We are now Homo Connectus:  Humans as one hive-mind organisation, with the knowledge of the entire civilisation available to each of us.  

Yes, I’m talking about the smartphone in your pocket.  Twitter.  Facebook.  Blogs.   24-Hour Rolling News.  It’s truly exhilarating to have watched this change unfold during my lifetime.    

For the first time, the whole planet is waking up as a collective consciousness. 

Or perhaps it shouldn’t be waking up – more like, being born.   We’re in the early years of learning to talk to one another, discovering the thoughts of everyone else, getting to grips with different viewpoints and opinions on an unprecedented scale.

We are only just beginning to learn to organise discourse.   Reddit, for example, is a site which allows anyone to create a category.  Then, within that category, discussions happen.   (Hacker News does the same thing for a single category).

We have issues with trolls, and confidence tricksters and fishing, and bullying and baiting.

Different viewpoints on things like racism, sexism, humour, politics cause vast amounts of discussion.  Not much of this is useful.  We need ways in which we can organise discussions better than a temporally-threaded web-page.   We need artificial intelligence to help us do this.   Actually, before that, we need systems that help us do this.

It’s really early days.  We’re at the level of infant-school playground right now.  But, we need new tools and techniques to help us get through the Eternal September.

comment 0

My Top 10 Silver Bullets

In the original “Mythical Man-Month”, 1975, Fred Brooks (who said many very insightful things also) posited:

“There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity"

Many years ago, I bought the 20th Anniversary Edition of “The Mythical Man-Month” and read the essay  ‘“No Silver Bullet” Refired’.  (So that would have been 1995).  In this, Fred talks about the potential Silver Bullet of re-use, and is cautiously optimistic. 

I think if he’d written this in 2005 (at which point Java was big news, and we downloaded libraries like Xerces and Tomcat from the internet) he would have said that this had been a resounding Silver Bullet success:  there just needed to be some other things in place first:  like the internet, open source communities, widespread adoption of Java etc.

So, why do people constantly go on about the idea that there are “no silver bullets”?  (Note that I’ve just recently watched this) There totally are silver bullets…  what Brooks wrote in 1975 is not necessarily true today.  

The rate at which technology was improving in 1975 was slower than today.  Given that Brooks’ time-frame was ten years, and there was but one silver bullet, you would expect us to be having more of them today:  ten years is a long time in IT now.  

But, there’s a second problem.  That is, they creep up on you, and become part of the scenery without you even noticing their contribution.   And, it’s really easy to underplay the importance of some massive ground-shift in opinion after it’s happened.  

In a way, this is like AI: as soon as a computer can do it, it’s not AI anymore.  Poor AI.  Self-driving cars, speech-to-text, text-to-speech, etc.  

Here are some silver bullets.   I’m sure there are better ones, feel free to suggest some.

1.  Assembly Language

It’s difficult to underplay the importance of having mnemonics (which imply a meaning) instead of just numbers.  Numbers you have to remember, and hold in your head.  Mnemonics leap off the page and give you a meaning.  It’s a pretty simple translation, granted, but it’s a huge, easy win.  This gives us a clue as to where to find silver bullets in the future:  look for ways in which we can make the task of defining software more human.

2.  Higher Level Languages

Suddenly, anyone can program, by typing stuff in and watching what happens on the screen.  No one has to worry about memory location, bytes, bits, that kind of thing.  Ok, it’s not suitable for everything but still, a huge win.  My favourite example is Excel.  No one even realises that doing Excel formulas is even programming.  You don’t worry about garbage collection, or deployment, or linkers.  But, it’s undeniably happening. 

3.  Library Code

By creating a language that allows libraries to work together, suddenly, the lone programmer doesn’t have to do it all herself.  (This was Brooks’ re-use option).  Also, we have google now.  And github.  I can type a few keywords in and instantly discover all the libraries in a given area.  I have to make a value-judgement on whether any of them are good, but still, I have the choice of not building it myself.

4.  Garbage Collection

So, you don’t have to worry about memory anymore.  Awesome.  A whole set of bugs rendered irrelevant.

5.  Type Systems

Stopping you breaking your memory models.  Catching loads of bugs.  These were probably around when Brooks wrote his book, but maybe wouldn’t have been widely used like they are today.

6.  The Internet

Instead of distributing libraries on floppy disks and charging a lot, you can now download them off the internet.   Also, you can easily find out about libraries and languages that are out there by searching for them, or reading newsgroups or whatever.  Gone are the days when I had to own a manual in order to understand something: I can just download it now.  

7.  Open Source

Not only can you find out how the libraries work, but if you find some like-minded people you can work together on solving a common problem.  A problem shared is a problem halved.  Or whatever.   It’s self-organisation on a huge scale.  

8.  Stack Overflow / QA Sites Generally

If downloading the manual doesn’t cover it, just ask a question and then for no money people round the world will figure out the answer for you.  I have no idea why, but it works.

9.  Commodity Hardware & Operating Systems

Brooks talked a lot in the 20th edition about the explosion in micro-computers.   He also talks about consolidation of operating systems, which has made a huge difference.   He would be amazed at what this has become.  Both those things are hugely more advanced now.  We have computers in our pockets.  OS X, Windows, iOS… they’re even free.

I was just going to put Linux here, and say how, we used to have to worry about getting files off one machine and into a format for another machine, and finding the right media to do it… that’s gone away.  Linux standardises at such a high level that you can assume so much about the platform you’re running on.

Also, we now even have cloud computing – you can just provision hardware in seconds.  It used to take months to buy kit and get it set up and connected to the internet.  

10.  Web Standards

To put my software in front of users, I don’t have to send them a CD with a client program on it anymore.  No, I just point them to a “URL”, whereupon their browser will download the client in “HTML”, “Javascript” and “CSS”, and, even though they might not be using exactly the same browser as me, it’ll run for them anyway, and they’ll see my software.

Even just the improvements to browsers over the last ten years have been huge.   Anyone remember programming for IE4 or 5?  


Brooks signs off the book in an epilogue called “Fifty Years of Wonder, Excitement, and Joy”.  And he’s totally right.  The silver bullets are coming thick and fast.  I expect to see golden bullets in my lifetime.  As he says:

“Not only is the end not in sight, the pace is not slackening.  We have many future joys"

comment 0

Constraints Make For Flexibility

Antione de Saint-Exupery said:

Perfection is reached not when there is nothing more to add, but when there is nothing more to take away.

Which I think, is one of my favourite quotes of all time.  It resonates on a deep level.  And, it’s extremely relevant to computer science, where we talk about YAGNI (You Aren’t Going To Need It) and Refactoring Mercilessly.  It’s also appropriate to the startup community, who talk about building their Minimum Viable Products.   That’s all true. 

Of course you shouldn’t build stuff you don’t need.  It’s waste.  And, waste lying round everywhere just makes everyone’s life harder.  Get rid of it.

But, those things don’t even capture it.  This is also about curation.

Sometimes, by limiting functionality, we end up with more.  Not just, the same, but actually more.

For example.  Pure4J disallows you from doing certain things with code. And, it enforces those rules pretty rigorously.  By limiting what you can do, suddenly, a whole new load of things open up.   Problems that you had before just disappear. 

Here’s another example that I’ve spoken about before:  Creating free-text fields in Jira.  By putting these in, you actually reduce the functionality of the software, because now there are plenty of ways in which information may or may not be stored in the system, and no one can rely on it being there.  By having a fixed, mandatory choice of four options (creating a constraint) you give people something they can use to categorize and search against.  But, with an optional free-text field, it seems like you’ve given them more, but actually, you’ve given them less.  It’s next to useless for any kind of searching or categorizing and may as well not be there.

A third option:  Rich-Hickey’s Datomic database.  It’s basically a layer over a regular database.  And, it prevents you from changing stuff.  So, you can only add things to the database.  How the hell is that better?

It sounds like it should be a disaster.

But, it isn’t.

By doing this, you can pass around copies of the database.  You can test in production.   You avoid a whole host of locking and atomicity problems that you had before.  And what have you really lost?   Actually, nothing you couldn’t do without.

So:  disallowing certain behaviours makes a system more flexible.  Don’t give users more than one way to do anything.  Just give them the one, right way.  

Can anyone think of other examples of this pattern in action?

comment 0

Buffet On Short-Squeezes

Reading Buffett today.  He said something amazing.  Talking about the rise of the auto industry in the US, he said (paraphrasing):

“It’s not always possible to predict the winners. However, it’s often easier to predict the losers.  I’m disappointed the Buffett family didn’t hold a short position in horses.  We could have easily borrowed some if there was a short squeeze”


In a short position, you are basically selling horses.  e.g. Owning -100 horses.    When the price goes down, you make money.  Then, you can buy more negative horses.  Let’s say a horse costs £100 today.  I sell 100, netting £100,000, but owing 100 horses.  In 3 months, the price declines to £90.  I buy 100 horses, learning the balance.  It costs me £90,000, leaving me £10,000 up.

So, to do this, I would have to borrow 100 horses in order to sell them.  (this assumes I can give back different horses… it would be different if these were specific horses).

Short Squeeze

So, what is a short squeeze?   This is where lots of people hold a short position, and the price of the horses increases.  Maybe it’s harvest time, or something.  In this case, the price might go up.  People with short positions might see their losses increasing and realise that they will go broke if they’re not careful.  

Brokerages have margin accounts.  This is money that you deposit with the brokerage to cover the losses of short-selling, if any accrue.  So, if you held a large short position, you might get a margin call on the account and have to give the brokerage more money. 

Let’s say the margin account balance was £5000.   If the price of horses increased to £105, your short position would be liquidated.  You would end up losing your margin of £5000, because you would spend the whole £105,000 buying back the 100 horses you owed.  (Ignoring options here, for simplicity).  

Otherwise, your short position would be closed before it was the brokerage rather than you that was accruing losses.

Borrowing More Horses

So, if the price increased, Buffet could borrow more horses.  And then, sell those horses too.  And pay more money into his margin account.  This would increase the size of his short position.  But, crucially, he would still be able to cover his margin calls.

He would then be paying borrowing costs on the horses (maybe £5 per horse-month?), rather than covering margin calls.  Arguably, this would be better – he would be able to “ride out” the squeeze.  But, if his long-term bet against the horse failed to pay off, he would be ruined.  

Possibly why his family didn’t do this.


     “The market can stay irrational for longer than you can stay solvent"

comment 0

Round-Trip Transformations

Following on from “Logging vs Breakdowns”, here is my second Banking pattern:

Apart from pure calculations, all other data manipulation should be round-trip transformations.


  • CRUD operations:  Store some data in the database, read it back. (Upsert/read or Delete/read)
  • Marshalling:  turn data into XML, and back again
  • Buses:  put a message on a bus, read it off.

Read-Only REST Service?

Let’s say you have a  third-party service publishing data.  To get it, you need to make a REST call.  Let’s say, it’s price information on some bonds.  

What are the chances that you have read the price information properly?    Well, Murphy’s law says that if you don’t code this as a reversible transformation, you’ll get it wrong.  So, how can you do that?

Simply, you need to extend the service, so that it is reversible: maybe you create a mock of the (read-only) service, so that you can put data in, and then read it back, and the reading part is functionally equivalent.  (i.e. using the same HTTP protocol etc.) In this case, you are testing all the code, but you are not going to use the writing chunk in reality.


A round-trip transformation is a homoiconic representation of the data.  It’s the same data in a different form.   As opposed to a calculation, where fundamentally we are creating something new out of the data.   It should be easy to tell where this pattern applies by asking the question:

Is this operation reversible at all?  Do I have everything I need to revert back to what I had at the start?

I.e.  given f(x) -> y, can f’(y) -> x exist?

If so, you are looking at a round-trip operation.

Write-Only Services?

In a way, it’s even more important to subsume a write-only service into a round-tripable operation.  This is because, you have no idea if what you wrote made it there at all.  Again, it’s essential to mock-and-extend the interface to ensure you are testing the round-trip.

Still Not Perfect

This pattern falls short of perfection: let’s say you have to persist fields A, B and C in the database.  But, on write you transpose A and B, and on read you make the same mistake.  On the database side, your data is wrong.  At your side, it’s always correct.  

Unfortunately, there’s nothing you can do about this. There is always the chance of this happening.  Essentially, you have written the code and the test, and the same error is present in both.   The best you can do is to try and write the read and the write separately, so that you are coding both in a different way.  Otherwise, still it’s tough.

I don’t believe there is a way to solve this problem.   

Notable Exceptions

The obvious one should be logging, but we already tackled this in “Logging vs Breakdowns”.  But, really you should still be logging that you are performing some manipulation.


Break down all the functionality in your process to either round-trip transformations or Self-Consistent reports.  It should be clear from this that data is always bulk-gaining as you go along.  

comment 0

Why Are We Not All Just Building Things In SQL Databases?

It should have been so easy:  transactional datastores, functional query languages on top of them, and then a programming language to modify stuff.  The promise of the DBMS.  But what went wrong?  Why do we have such a messy, complex landscape?

One problem is the cost:  databases used to be hideously expensive.  A second is the language:  programming in SQL sucked.  And while things like Oracle tried to bolt on Java as best they could, this always seemed… messy.   Coding everything with stored procedures didn’t work. 

And then there was the web:  Databases were a poor option if you wanted to build interactive web pages, whereas servlets gave you the freedom to do anything that the protocol supported.  So, really there was no choice.  In order to build web-enabled applications, people were forced outside the DBMS to deliver them.  

Another thing is separation of concerns:   If you build outside the database, then maybe you can swap the database for something else, at some future time.   This never happened, in reality, but at design-time it gave you extra choices.

What about in banks?  Going back, I remember the GriMis database at DB: you put risk numbers in, and they were pulled back into the report server.  It had to be this way as the databases weren’t fast enough to do the reporting themselves.  So, again, the databases let the side down.  Otherwise, you could have achieved the same result doing just SQL, and we could have saved the man-years of effort building the report server first in C++ and then in Java.

Somehow, over time, databases got relegated to the role of just being data stores, and application servers took over creating transactions for them and hosting the applications.   Somehow, this was faster (despite extra networking costs).  And, this allowed databases to scale horizontally into the unmanageable mess of one-for-each-app rather than just vertically one-for-the-whole-enterprise which was bound by the size of the DBMS architecture and the hardware it ran on.

It’s not hard to see how you go from this fragmented picture to where we are today:  in each case, the short-term advantage of building something now trumps the long term homogeneity of the environment.  And so, we end up in massively heterogeneous environments: there is no long term.  The only place that homogeneity exists is at the network layer.    


  • Heterogeneity wins, because decisions about complexity are made considering only the short-term.  
  • Heterogeneity wins, because, having given up the long term, people optimise on the problem areas they are facing, rather than considering the big picture.
  • How did homogeneity win at the network layer??  There were standards, they bought the same wires for the whole building, maybe the problem domain was small enough that they could consider it all in one go, also there are ways to convert between one network and another.
  • Do One Thing Well.  You can’t possibly solve all problems in the database.  You need to go wider.  So, programming languages were always going to be a separate thing to the database layer.  They couldn’t own the whole picture.
comment 0

Logging Vs Breakdowns (A Reporting Pattern)

Recently at work, we have been discussing our risk flow application, and discussing how logging should work.  The question is:

When a business-user has an issue with the report, how will we log the details and information used to construct the report?

Discussion has centered around a standardized logging format, or a standardized location for where this logging information should go.  However, I think all of this misses the point, and instead, you should introduce a pattern, the Self-Consistent Report Pattern.  I will discuss this in detail further down, but before I do, let’s look at…

What’s Wrong With the Logging Idea

The problem is this:  The business receive report A.  However, they are concerned that report A is wrong.  This means that either:

  • the data coming into report A is wrong, or
  • the calculation performed in report A is wrong.  

Obviously, now they have to investigate further.  So, the next step is to get hold of the aforementioned log files.    Now, maybe this means a call to the IT team.  Or, maybe there is a service that the logs can be pulled back from.  Either way.

Having got hold of the log files, they then need to parse the log files looking for the information related to the error they found.  Maybe there are four or five items of information that come in, and they have to find each of these in the log file.   

Next, they can use this information to try and perform the same calculation that was on the report.  If they come up with the same result, then they know something must be wrong with the input data.

If they come up with a different result, then they know, something is wrong with the calculation.

One final thing that could happen is that they realise that actually, they were wrong, and in fact, there’s no problem.  But, excepting this, they can then go write up a bug report.  

So, what’s wrong with that?  Multiple things:

  1. The business user now has two separate sets of data coming in.  The report and the log.  There’s a potential for error here:  what if he gets the wrong log?  What if the log is deleted?  
  2. Logs tend to be full of stuff.  The business user doesn’t care about all the stuff to do with SQL queries,  heartbeats, message queues.  They just care about the input data and the calculation.  So this stuff obfuscates what’s needed.
  3. There’s every chance that the developer hasn’t included what’s needed in the log file.  In which case, nothing can be done.  Is there any way that we can ensure the log contains everything needed?  Possibly not.
  4. Once the business user gets just the information he’s interested in, he still has the problem of tying up this information with the information in the report. Maybe that means lots of data merging and pulling stuff into Excel.  Either way, that’s big bit of work to do, and a likely source of error, even before he gets to the calculation.

So how can we fix this?

The Self-Consistent Report Pattern

Let’s solve this problem by adding a single constraint to our report format:

“The report should contain both it’s input and output data.  The output data should all be derivable entirely from the input data."

So, this is to say that if we have a calculator that (trivially, here) sums up values A, B & C into a result R, then A, B, C and R should all be on the sheet, like so:

5 9 4 18
2 11 2 15
1.5 8 4 12.5

At a stroke, this has solved problem 1 and 2 and 4 we identified above:

  • There is no need to get a separate log, it’s all here (1).
  • Done right, the report contains only the information you’re interested in, and not every single attribute of data ever loaded (2). 
  • There’s no need to do the matching manually, it’s done for you (4).

What about problem 3?  This is something you can verify by loading the report into Excel and figuring out whether it’s self-consistent or not:  you get some business user to try and validate the results, based on the information that’s given to them, and nothing else.

But obviously, if you go this far, you could just automate that test and write an acceptance test that makes sure that the results on the sheet can be calculated, so the business user doesn’t need to do it over and over for each release.

The Self-Consistent Report Pattern: Reloaded

You can go one step beyond this, and actually add your testing code into the report itself.  That is to say, the system responsible for producing the report is also responsible for testing the consistency of the report before it is published.  Essentially, you are building the acceptance test into the report itself.  

This last idea works especially well when the report is at multiple levels, or roll-ups (Maybe a trade level and a counterparty level, or line-items and totals).   You might produce these two distinct levels in different tabs of an Excel sheet, or in different files, but you can run the consistency check across all of the levels, to ensure the counts match up, or the totals match up to the line-items.  

comment 0

Visibility:  The Greatest Loss

I am starting to believe that the reason programming is difficult is because of the visibility issue:  There is no relationship between the code you type on the screen, and what comes out of the other end.

It requires a strange imagination to bridge the gap from one to the other.  Not to mention an overwhelming desire or need to get something done.

The closest that the man in the street gets to programming, is with Excel.  Even Excel ruins visibility.  It’s really easy to lose track of how a calculation works in Excel: there’s nothing preventing that happening.  How is this cell calculated?  Well, I can see the names of the cells being referenced. If I click, I can see them: the important cells get highlighted in rainbow colours and I can see what connects to what in the formula.   But, I have to interact with Excel to get this to happen:  the model is built in my head, through my exploration of the sheet.  I am not shown the model, I have to construct it myself, cell by cell. 

Visibility is the most easily-destroyed helpful property of software

It’s not even something that’s really mourned after it’s gone.  The visibility remains in the mind of the developer.  Just like, if I hide a ball behind a curtain.  There’s no loss to me:  I know the ball is behind the curtain; I put it there.  The loss is to time, when the ball is forgotten, or to the next person, who needs to know where I put the ball.

But, if we want to create systems that can be understood then visibility is really really important.  

Banking Risk

If you are building a computer game or a website, then this is not a problem that affects you too much:   it’s all on-screen.   Arguably, you can’t just eyeball in either case:  to see some features of the game or website you’ll need to interact with it.  So, there are things that are invisible until you explore.

But, in the world of banking risk, everything is invisible.  Numbers are completely meaningless until you introduce some visualisation.  But, visualisation is expensive.  Yes, you can put some numbers into an Excel spreadsheet.  But it’s still quite a lot of work for IT guys to take numbers and display them meaningfully.

Wilful Loss Of Visibility

Sometimes, people deliberately want to obfuscate the logic and meaning in order to build privileged positions for themselves.  I see this happening at work, but Michael Lewis does a good job of representing this in Flash Boys too:  people profit from obfuscation:  In the land of the blind the one-eyed man is king.


Abstraction is a key feature of most programming languages:  the ability to introduce an indirection to say:  these cases are all examples of a more general case, so use the general case.  But, as soon as we have this indirection, we lose visibility over what is actually being done.  This is the cost of abstraction. 

Indirection means that my feet can operate the pedals in the car, but I don’t know what is actually happening after that.  


The mere idea of encapsulation is an affront to visibility:  you are saying:  “This here is a machine.  You don’t need to know how it works inside.  Use the interface and it will keep you from doing anything with the machine that you’re not supposed to do”.  

But, this is like my washing machine at home.   I have literally no idea of what’s going on inside it.  I put the clothes in.  I press some buttons.  There’s water involved.  Clothes come out clean.  

Visibility is also the cost of encapsulation.  I can interact with the device through its simple interface.  But, as a result of providing me that simple interface, the manufacturers have (possibly deliberately) hidden from me what goes on inside.

Modern Software Techniques Are Failing Us

These tools:  encapsulation, abstraction, are useful.  But, they don’t come without cost.  Can we not have tools which allow us to work without this cost?  

Part of the problem is the map and territory issue.   When we design software, we end up with a bunch of bytes on a computer disk, that make sense to the computer.   They make no sense to us.  People don’t work this way.    When the programmer creates the program, he is primarily concerned with communicating to the computer what it needs to do.  But then, we’ve added layers on top to make it easier for the programmer to use his mental faculties over time.  Things like, assemblers, third generation languages, object orientation. 

The Real World

But this is analogous to the real world: “Stuff” is just atoms.  And atoms don’t make sense to us.  Our senses are all about managing the signals produced by these atoms, and converting them into something upon which information processing can be done.  

From this, we develop mental models of things:  mental maps of territories, mental models of people’s behaviour.  Mental feedback loops: “if I eat this, I will feel full”.  

In the real world, people have to do lots of work to build maps, guides, teachings that other people can use to understand the world in the way they do.  

If we conceptualize a computer program as a foreign land, it’s almost as if the programmer is primarily sculpting this landscape, but not producing any maps for it: people are forced to learn the territory themselves, or treat the whole thing like the washing machine and not worry about how things happen. 

Brain-Computer Interface

Tools (like Eclipse, and Excel, say) don’t just try and reduce our finger-typing.  They are a brain-computer interface.  In order to fully exploit the pattern matching and inference-building techniques in the brain (the uniquely human talents which we don’t yet understand how to build in software), we need to increase the surface area between the human and the computer.

Since we can’t change human perception (yet), this means that software has to adapt to exploit the human perceptive system.  This means, on a basic level:

  • Using colours (e.g. to highlight our code)
  • Using indentation to give structure (geography). 
  • Using visualisations where possible, instead of tables of numbers.
  • Using words and iconography over pure opcodes.
  • Graphs and Maps at different levels of abstraction / scale.
  • Domains:  subdivision of the space so that you only need to consider parts in isolation.
  • Sound? 
  • Smell!?

Obviously there are areas that programming doesn’t exploit well at the moment:  geographic structure is really really poor.  Programs and data are often represented in trees (or, hierarchical structures).  These trees tend to be canonical, in the sense that they are one particular tree, rather than a choice from all possible trees.   For example:   java classes are shown in a package structure in Eclipse.  However, you could show them in an import-dependency structure.  This wouldn’t then be a tree, it might have cycles in it.  

Machine Code & Beyond

What something like assembler language does, is transform machine code into something more visible and human.  Abstract numbers become instructions.  We have removed (or at least, created a simpler) layer of indirection.  This improves visibility:  we are trained to recognise words and their meanings.  So, assembly language is the first example of improving visibility via the BCI techniques described above.

After assembly, we went beyond this:  Java, Lisp or Haskell (or any 3GL) use keywords profusely, and you get to define your own lexicon.  

Further, Java, Lisp and Haskell all take different approaches to reducing the problem space.   Java uses object orientation to reduce the scope of what you have to visualise in one go.    This is again another concession to the human visualisation system: "we know you can’t visualize anything, but you can visualise some smaller domain”.   Lisp uses its single list abstraction.  Haskell uses pure functions.  

Excel uses both pure functions and geography.  

None of them are particularly rich in higher-level abstractions.  Java for example has packages, which are fairly weak: you can create all kinds of inter-dependencies between packages (and the objects within them) which destroy any kind of coherence  / loose coupling that the package has.


If we want to build new, better languages, and improve the way we do things now, we need to take the hints from nature.  Obviously, this table is wrong, but it serves to indicate that we have a lot more to do:

Life Level Computing Level
DNA / Proteins / RNA / Amino Acids Machine Code / Bits / Bytes
Organelles / Viruses Functions, Methods, Data Structures
Cells Objects / Source Files
Organs Packages ?
Phenotypes Programs
Families Operating Systems / Servers
Societies Organisations

Questions Arising

  • If these different levels of hierarchy are so useful, why can’t you define more of them?
  • Pure4J shows that you can do a lot with annotations.   Could we also have annotations to describe this kind of super-structure in Java, say?
  • What benefits would that give us?
  • We currently deploy and run stuff at the program level.  We are starting to move up to the next level with things like Docker (i.e. OS/Server level).  But this is hopeless when we have a whole Bank to change!  What would it look like if we could manage everything together?

comment 0

Response to Agile Principles

"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."

  • Is this satisfying the customer?
  • Does the customer always want software continuously delivered?
  • Do they want it early?
  • Is there such a thing as too early?
  • I think the customer would often happily trade off “Early” for “some semblance of finished”.  Early means re-testing.  Early means trying to use something with features you need missing.

"Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage."

– Is there actually just one customer?
– We should be working out ways to prevent requirements changing, and also to elucidate requirements early on.  
– Often, the requirements are only really apparent after some software has been written.
– But, often the requirements aren’t there because people haven’t sufficiently thought them through.
– There is no reward for coming up with decent requirements early on.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

– The reason for this is not get value delivered early on, per se.  It’s to get feedback loops.
– Also, the reason is that by practicing deployment early, you get good at it.
– But, there are hidden costs to deploying software (even with CI/CD).  People want to check it.  There are often forms to fill in, security reviews to do, and so on.   
– Software with only a few features might not be worth using at all.

Business people and developers must work together daily throughout the project.

– Except this happens never.  If they spent all their time with developers, they wouldn’t be business people.
– This is the single biggest failure of the agile process.  Business people (the people you need to help you) are always too busy to spend much time on the project.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

– Invariably, no project is an island.  Often you have to interact with system B, etc.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

– Developers hate talking to each other.
– People are often working in other countries, in other time zones.
– Face to face conversation is OK, but you need to write things down so that you remember what was said
– Often, everyone disagrees.  You sometimes need sign-offs.
– Lists are really useful.  
– Conversations are lost the moment they are complete.  No one ever remembers conversations the same.  Often, important people are excluded from conversations. 
– I conclude that this is very inefficient and very ineffective. However, it still might be for some use-case the most efficient and effective.  This is a shame.  
– Open source software gets built with almost none of this.  Explain.  
– Groupthink.
– Design By Committee. 
– There are clearly more productive ways to develop ideas. Our brains somehow do it (organising the thoughts of billions of neurons).  This is why art is a one-person-show.  If only we could access these organisational constructs.
– Right now I am in an online meeting.  We have a list of actions.  We are taking notes.  We are assigning blame.  We are capturing issues in a list.  It’s all kind of ad-hoc.  None of this is face-to-face communication.   There is a continuum here of online, offline, ad-hoc, recorded, signed-off, tracked, untracked communication.  Why?

Working software is the primary measure of progress.

– Which is measured how, exactly?
– What constitutes working? 
– How do we weigh the software we have working?
– Maybe this is counterproductive.  Less software is better than more.
– Bill Gates: “SLOC is like measuring completion of building aircraft by weighing it”.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

– This isn’t even a principle.  It’s just two sentences saying what you hope will be true about the world.

Continuous attention to technical excellence and good design enhances agility.

– I think this is another way of saying, get rid of technical debt.  

Simplicity–the art of maximizing the amount of work not done–is essential.

– I think this is another way of saying, get rid of technical debt.  

The best architectures, requirements, and designs emerge from self-organizing teams.

– If this is true, why do we need agile to tell us how to organise things then?

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

– This is really just the Lean/Deming/ISO9000 thing all over again.  Is this even true anymore?  Why do teams only concern themselves with their own processes?  Can’t teams learn from each other?  Why does each team have to re-invent the wheel.  
– Isn’t software supposed to be some unique thing?  Shouldn’t things change as they go along?