Microservices In Embedded Systems

I don’t write many “technical” posts, finding people and organisations more challenging, and hence more interesting, but this is one. If you like the diversion, let me know and I’d be happy to write some more such posts.

Some years back I was running a boutique embedded systems development company, in partnership with a hardware engineer. He’d design and build the hardware – micro-controller based – and I’d implement the core functionality in software. Mostly in C, C++ or Python, with some e.g. PIC assembler here and there. OSes were Linux, WindowsCE, or naked. Approach was entirely Agile (on both the hardware and software fronts).

We specialised in hyperconnected devices – now more widely known as the Internet of Things – so every device we designed centred around an embedded web (HTTP) server.

One particular project – at the higher end of micro-controller capabilities, with virtual memory addressing and a fairly beefy CPU – was an event logging device with e.g. remote access (dial-in or ethernet) capabilities, a touch panel for local admin, and a bunch of digital I/O lines for scanning the connected devices to be logged.

I chose Python as the prime option for the implementation, with Javascript and CSS for the touch UI, some C for low-level hardware (device driver) functions, and Json for initial device configuration a.k.a. policies.

Once I had a Python implementation that would run on the platform, I gave some thought, as I was writing some early code, to a suitable architecture. I knew the client had a vague interest in extending the range of product variants in the future, as well as possibly adding more features, so although not a priority, extensibility was at the back of my mind.

Aside: Another reason for Python being the favoured implementation language was the excellent CherryPy webserver package, which I had used often before. Unfortunately, it turned out that CherryPy wouldn’t run on the platform (WindowsCE), and tight schedules did not permit an investigation as to why. So I went with a basic SimpleHTTPServer implementation instead.

More pressing, in terms of getting the whole thing working, was the memory usage of each Python process. And the relative shortage of memory on the device (suggested by the economics of the design). This consideration was paramount, it turned out, and a good deal of the work was a consequence of shoe-horning the necessary functionality into the memory space of the device.

Eventually, the architecture evolved into circa a dozen separate, cooperating Python processes. (Note: I stuck with Python, because actually implementing the necessary functionality in Python was much quicker and less painful than doing the same in e.g. C or assembler).

In retrospect, today we might choose to call this a microservice architecture. Each Python process ran independently of the others, doing its small bit, and interacting with one or more of the other services as and when necessary.

The microservices included:

  • Watchdog
  • MCP
  • Networking
  • TimeWarden
  • MediaWarden
  • EventScanning
  • EventLogging
  • Webserver
  • RemoteCommandListener
  • SyncSource
  • DeviceStatus

Aside: Actually, there were rather more microservices than this originally, but I had to combine some in order to meet the memory constraints.

At the end of the project, it was clear (to me, not to anyone else) that, serendipitously, the architecture was such that adding new features, even whole new microservices, would be a piece of cake. I’m not aware that this was ever appreciated, let alone acted upon (no one else involved was at all software savvy).

Have you worked in embedded systems development? Have you ever seen or considered the pros (and cons) of a microservices-style architectural approach?

– Bob

Further Reading

Stop Thinking About User Interface ~ Juan José Ramírez

What’s In It For Me?

The general election is all but over here in the UK. It’s been marked, as usual, by incredible promises and naked appeals to self-interest from the major parties, and incredulity and apathy from the electorate.

It seems the political elites have a profoundly Theory-X view of the voters. According to the political rhetoric, everybody is out for themselves, venal egotists. Everybody is assumed to want more money, less time working, more pandering to their biases, and an easier life in general. Psychiatrists call this projection.

Whilst I can believe that many in politics are all out for themselves, I daily see evidence that people in general are much more social. Much more interested in what’s in it for other people. Much more concerned for the communities in which they play a part.

But this isn’t a post about the election. It’s a post about how the workplace – and the way the work works – often places people in situations where they feel they need to look out for themselves, rather than follow their natural, human inclination of looking out for others, too. I’m pretty sure if we had a different political system, one which encouraged politicians to attends to folks’ needs, rather than their own, we’d see a different kind of politician, a different kind of politics, and an electorate much more engaged with the whole process.

And isn’t that what we’d like to see in the workplace? A different climate, where people engage willingly with each other and their common challenges, and look out for each other in getting things done?

And just like in politics, I see little prospect of things changing markedly any time soon. But, just like in politics, we don’t have to play “their” game. (Although it’s not their game so much, as the game imposed by the system).

Would you be willing to take up my invitation to consider how you could change you workplace for the better by bucking the system and giving others, and their needs, even a just little extra consideration?

– Bob

How to Develop Software – And Why Most Organisations Can’t

There’s a simple recipe for developing cool, awesome, high quality, cost-effective and just plain sexy software. Most developers know this recipe. Most organisations do not.

How To

Get a (small) bunch of people who enjoy working together, who choose to work together, who choose what they’re working on and the scope of that work, who choose their own tools (if any) and equipment, who love what they do (and why they do it / who they do it for), and who have the autonomy to figure out how they can work best together, in the longer term. Offer them them some difficult (but not impossible) challenges and provide them with the wherewithal* to get on with it. Invite them to deliver working stuff (into production) frequently, and make sure the rest of your organisation knows that everyone benefits from immediate and meaningful feedback each time they do that.

(*By wherewithal I mean information, money, connections, someplace to work, and support).

Why Most Organisations Can’t

Most organisations cannot do this. They cannot attract the kind of people who can and want to work this way. Even if they could, they cannot trust these people to just get on with it. Even when seeing demonstrable progress every week or two. They cannot tolerate the absence of any “management” roles, whether this be overt (e.g. line managers, project managers), or covert (Product Owners, Scrum Masters, et al.). They cannot tolerate people “doing stuff” that they don’t understand, in ways they don’t understand. They cannot reliably give immediate and meaningful feedback. Above all, they cannot tolerate people having fun. After all, work is a serious matter, and results count for much less than how they’re arrived at.

Bonus Tip 1 – What To Do When Your Organisation Can’t

Make your organisation one that can. One that can attract the kind of people who can and want to work this way. One that can trust people to just get on with it. One that can tolerate the absence of any “management” roles – rejoice, even. One that can tolerate people “doing stuff” that they don’t understand, in ways they don’t understand. One that can reliably give immediate and meaningful feedback. Above all, one that can tolerate people having fun. After all, work is a serious matter, and results count for much more than how they’re arrived at.

Bonus Tip 2 – How To Scale

If you have more than one piece of software on which you want to see progress at the same time, get two, three, or a hundred more (small) bunches of people. Ensure they have the wherewithal (including a common purpose). And let them figure it out as they go. Most organisations can’t do this, either.

– Bob

Agile Competency Is A Crock

LearBookOfNonsense

Part 1 – The Lede

The Agile Manifesto set out to make developers’ (and others’) live richer, saner and more fulfilling.

A true irony of the legacy of that Manifesto is that finding a fulfilling job or role “in Agile” is nowadays next to impossible.

Competency is not something valued by hirers and their gatekeepers. Being a “safe hire” is all.

Part 2 – The Background Story

My dear friend, the late Grant Rule, had many compelling stories to tell.

One of these concerned a large insurance company in the home counties. Let’s call them InsCo. For some reason, the powers that be became interested in the reasons why they were not doing as well as they thought they should be, business-wise.

Some number of investigations were commissioned. One concerned the type of people they were hiring, versus the type of people needed for business success.

To cut a long story short, it became revealed to them that not only were they hiring people with little to contribute in the way of the organisation’s business goals, they were actually hiring people whose general style actively undermined those goals.

In other words, their hiring practices were expressly filtering out those people best suited to make a positive contribution inside the business. And this had been going on for years, if not decades.

I always found the story fascinating, not least for its compelling ring of truth.

In todays’s business world, I see many of the organisation I visit or work with making exactly the same error.

Organisations whose hiring practices filter OUT exactly those candidates who would best contribute to the espoused goals of the organisation.

Guided by the heuristic of POSIWID, I assume that organisations – or more exactly the core group within an organisation – are not much interested in the organisation’s espoused goals. Deming said as much fifty years ago, with his First Theorem:

“Nobody gives a hoot about profit”

~ W Edwards Deming

I find this particularly noticeable in hiring for so-calle Agile positions and roles. […]

Now, I’m not about to criticise folks – senior executives and middle managers in this case – for acting in their own individual and collective (core group) best interests.

It’s what humans do – acting to get needs met.

I’m just inviting you, like the executives at InsCo did, to take a look at the consequences of your current hiring and staffing policies and processes.

And consider how those staffing policies and processes play against the things that matter to you.

Oh, and maybe consider what those things that matter to you are, too.

Part 3 – The Dilemma

For me, struggling as I am to find gainful and meaningful employment, the questions aired in part 2 raise an interesting question for all of us in the Agile field:

Do we concentrate on appearing competent, and on our abilities to help the organisation achieve its espoused goals? Or do we focus on getting a well-paid job – which demands a very different strategy and “personal brand image”?

The former strategy suggests we list our experience, results and contributions to the success of the organisations we have worked with. That we take hiring organisations’ espoused goals at face value and play to those declared goals.

The latter strategy suggests we present ourselves in terms that appeal to the needs we imagine the hirers – and their gatekeepers – have.

Needs rarely articulated and only determinable through observation of these folks’ actions. Needs which in most cases means portraying ourselves as conventional, conservative, and status-quo loving. As “safe hires”.

I’ve discovered – unsurprisingly, to me – that I just CAN’T bring myself to do the latter.

I’m NOT a safe hire, not do I ever wish to be. My value proposition is other.

Outwith the emotional consequences of pretending to be something I’m not, and setting myself up at work to live a life that’s a bald-faced lie, I just don’t want to find myself in any more jobs or roles that, in essence, are just another stupid punt.

How about you?

– Bob

Out Of House FlowChain

When I conceived of FlowChain, some six years ago now, my immediate context was development shops with their own in-house developers, and other supporting staff.

But it strikes me that with just a few adjustments, it’s also suitable for organisations that sub-contract out most or all of their development projects to third parties.

These adjustments centre around arranging for the various third parties (assuming, in the likely case, that there’s more than one) to each contribute staff to the “Pool” (see diagram). These arrangements include:

Commercial

How will the third parties be paid? Some UK government functions use function points as a measure of “work done”, with a set price for each function point “delivered”. See: Output-based contracts. I can imagine other contractual arrangements, too.

Social

How will the third parties’ staff integrate or form healthy relationships with the in-house commissioning staff (a.k.a. product owners)? Will there be shared spaces? Regular visits to and fro? Some more technical forms of communication (Twitter, chat, video conferencing, etc.)? Remember, each backlog item is sized for two to three people working together for two to three days.

Tooling

Third parties remain free to pull items from the backlog as they see fit (just as with in-house FlowChain), and use their own tools, languages, etc.. I foresee some advantages in having a common code repository, coding and other standards, agree requirements around test suites, and so on.

Delivery Into Production

Maybe the organisation contracting the third parties has its own Ops department. In this case the interface between development (teams, third parties) and Ops would probable best be standardised and agreed (like an API). If the third parties have their own Ops folks, then they can do DevOps in their own space and time, and serve the “production” services – or even micro-services – they each operate, directly to users.

Shared Backlog

For clarity, this variant of Flowchain retains the enterprise-wide backlog, with user stories, improvement stories, etc. being prioritised by some black box (or white box) prioritisation algorithm, committee, manager, or whatever. The only real change is in how the Pool is constituted. Note: I see no particular reason why the general FlowChain principle of “ANY unassigned development folks from the Pool can coalesce around each new top backlog item” cannot stand, here.

There may even be emergent advantages in having e.g. developers from different third parties coming together to collaborate on specific backlog items. How weird would that be? Again, policy would guide folks’ actions here.

Who would “manage” the backlog?  This could be done by a small in-house staff, or itself subcontracted out to one or more third parties. Note: the backlog in FlowChain is largely self-managing, in any case, given an effective prioritisation algorithm or approach.

Flow

Flow (of e.g. software into the hands of those whose needs are being attended to) remains the key objective of the whole approach.

Growing An In-house Capability

For organisations without an in-house development capability, this approach provides a simple(r) path to establishing and growing an in-house development capability. In-house developers can be added, one by one, as and when circumstances (budgets, priorities, etc) allow. These new folks can work alongside – and learn from – third-party staff already used to Pool working, and the balance between in-house and out-of-house staff, skill sets, etc. adjusted dynamically as needs dictate.

Drawbacks

The key drawback I foresee is in the matter of dev-ops integration (DevOps). This could prove more difficult, in the case where developers, etc. are out-of-house and Ops in-house. This seems a special case of the issues of outsourcing and offshoring, generally. But I’m sure a bunch of smart developers and smart ops can work this out, especially with some help and guidance.

– Bob

As Simple As Possible

As I have written about and explored the Antimatter Principle over the past two years, I have learned some things about how to explain it. Looking back through the collection of posts, and bearing in mind others’ reactions, I suspect I have done a fair but no way remarkable job of explaining why “attending to folks’ need” is the only thing we need to nurture awesomely effective knowledge-work businesses.

Accessibility

I guess Einstein really was a genius, not only for his ideas, but for the ways in which he made those ideas accessible to the general public. For example, regarding the Theory of Relativity:

“When you sit with a nice girl for two hours you think it’s only a minute. But when you sit on a hot stove for a minute you think it’s two hours. That’s relativity.”

The Antimatter Principle follows a simple line of reasoning:

  • Collaborative knowledge-work always involves humans and social relationships.
  • We find great joy in helping each other in meaningful ways.
  • The effective workplace is one which recognises and works with, rather than against, these facts.
  • Self-strengthening “virtuous circle”: As we experience the joy of helping others, we want to do more.

 When you sit with someone for hours helping them get their needs met, it doesn’t even feel like work. But when you sit working for just a few minutes disconnected from people, relationships and a social purpose, it can feel like eternal damnation. That’s the Antimatter Principle.

– Bob

Further Reading

Social: Why Our Brains Are Wired To Connect ~ Matthew D.Lieberman

Stuck

The world of software development seems stuck. Have you felt that stuckness yourself? The kind of stuckness where we find ourselves experiencing a difficulty and every attempt to get ourselves out of it only serves to maintain or worsen the situation?

This recent blog post by Steve Chapman (courtesy of John Wenger) explains the idea in much detail. In case you don’t have the time to read it in full right now, here’s a summary:

Stuck situations arise through the mishandling of difficulties in a number of different ways:

  • Trying harder from the same mindset that created them
  • Oversimplifying or denying the complex nature of the difficulty
  • Creating utopian oversimplified solutions (a silver bullet)
  • Accidentally creating a stuck paradox by attempting to resolve things from the same level of abstraction that caused the difficulty in the first place

“The world that we have made as a result of the level of thinking we have done thus far creates problems that we cannot solve at the same level as the level we created them at.”

~ Ram Dass

First Order Change

“First order change” attempts to resolve a difficulty from within the frame of that difficulty – an approach that, at best, results in some incremental shift but essentially only leads to more of the same. Software development has been stuck in first order change – with its emphasis on process, standards, technical practices, compliance, etc. – for close to fifty years now.

Second Order Change

“Second order change” is movement or action that attempts to resolve things from outside of the frame of the difficulty. Second order change interventions typically seem counter-intuitive, spontaneous, bizarre and experimental – the opposite of what we might call common sense.

The Antimatter Principle

The Antimatter Principle is an example of second order change. It steps outside the frame of the difficulties and the stuckness of the software industry, and into a different, new frame.

This new frame seems counter-intuitve: how can it possibly make sense to forgo transactional relationships with people and attend to their needs instead? Wouldn’t that just mean we’d be taken for all we have?

It seems spontaneous: attending to folks needs as and when they arise.

It seems bizarre.

It seems experimental. Where’s the proof? The evidence? The data? Why have we not heard of this in practice? Who has been doing this already?

It seems to run counter to common sense.

Typically, (first order change) interventions try harder from the same, outdated and stuck mindset that they are trying to alter, or over-simplify the problem by denying the ongoing, dynamic, complex nature of organisational life.

The Key Question

In his aforementioned post, Steve Chapman asks the key question:

What might a global, second order intervention – to totally transform our approach to change and development – look like? What counter-intuitive and perhaps seemingly nonsensical approaches would need to be bravely adopted? What difficult beliefs would we need to gently let go of in order to challenge stuck habits and experiment with new ideas?

Having studied this question for the best part of the past twenty years, the Antimatter Principle is my answer. And the difficult beliefs we need to let go of are those implicit in the Analytic mindset.

“Our individual and organisational stuckness seems rooted in our habit of trying harder to become something we are not [e.g. cogs in a machine], rather than slowing down and becoming more aware of what we already are [e.g. human beings].”

~ Steve Chapman

– Bob

Further Reading

Is The Antimatter Principle Useful? ~ Ged Byrne

 

 

Follow

Get every new post delivered to your Inbox.

Join 12,059 other followers

%d bloggers like this: