My thoughts about doing some of the spadework at unit testing time are relevant here. If per-function figures can be obtained through an enhanced, automated test harness, then normal regression testing will produce fresh numbers each time. There remains the call-tree analysis, but that could be automated too (with some initial manual input for function pointers, which shouldn’t need to be modified too often). These are my current thoughts and, like you, I welcome other people’s.

One thing many engineers overlook – the problem that as software is refactored, debugged/fixed, enhanced, etc. its RAM usage (stack in particular) changes. Maybe now we’re using more stack in certain places, maybe now we’re using less. **But no one ever goes back & re-evaluates the stack usage.**

I’ve worked in (with) a lot of different organizations, and I can say confidently that very few of them do very detailed stack usage analysis in the first place… and even those who do tend to do it one time, during the initial development, and then the subject is never re-visited. Typically task stacks (in the case of an RTOS) are sized conservatively, and then some kind of margin is added on top (either a fixed amount or a percentage).

My point is that the holy grail would be some way to analyze the worst case at build time, and then have some type of stack size configuration file generated which would be guaranteed to be sufficient / correct. This file would be used at initialization (when tasks are created) to ensure stack sizes are adequate.

This is unattainable, however, because the dynamic nature of most programs makes this difficult or impossible. For example, with function pointers, the call flow isn’t known at compile time. And with C++ and dynamic types (not to mention 3rd party libraries, etc.) the problem expands even further. Now adjust your compiler options, inline (or un-inline) some routines, get rid of some globals & pass more parameters, and bang! everything has changed again. Change an iterative algorithm to a recursive one, yep that’s gonna screw things up too…

I guess my point is that this is indeed a complex problem and I haven’t found the proverbial silver bullet. I just now went over to Nigel’s post & read it – he identified many of the same issues, especially the points about function pointers & also not measuring once & then forgetting about it.

I’ll be interested to see what others have to say. Although I’m a big believer in doing the required rigorous work each time, what I’d really like to see is some sort of automated process that (mostly) removes the human from the equation (in other words, a system that always ensures by the time the code is loaded on the target, we’re “guaranteed” to not have stack overflow problems). Not saying it’s attainable, but it’s an interesting thought. Until then, I think we’ll have to continue to use things like stack checking hooks, MPUs, etc. to detect the problem post-facto & recover.

Dave (and others), you might find this post interesting, though my comment underneath it is more relevant to your point about timescales:

http://embeddedgurus.com/stack-overflow/2011/08/rabbit-patches-and-embedded-systems/

In my experience, the one thing more than any other that contributes to poor quality is when management insist on development time scales that are too short and thereby result in the job being rushed and botched through. The very fact that we are using software in an embedded system means that the problem is more complex than can easily or cost effectively be solved by hardware alone. Therefore this complexity needs a commensurate amount of engineering time! The only way that I can see to deal with this problem is the practise of software estimation, even though this is just as difficult as Peter’s experience with high reliability hardware design and often just as tedious. However, without it the software engineering team has no stick with which to bat back the big stick of management!

Hear, hear! I wish more of our peers would take software quality seriously. Over the years, I’ve learned a lot by working with people who design firmwarethat is life-critical – peoples’ lives are saved when it works, and people die when it malfunctions. I try to apply the same principles of every system I work on, even if lives aren’t at stake.

I’ve found that shops that “test bugs out of the system” tend to have low quality. By “test bugs out of the system”, I mean using the test phase to find bugs that they pretty much “know” are in there, because of deficiencies in specification / design / implementation phases. In other words, they enter the test phase *expecting* to find bugs. If the test phase was treated more like a verification phase, the expected outcome would be different.

In probably 3 or 4 opportunities in my career, I’ve encountered someone who acknowledged a design deficiency, but basically said “unlikely to happen, and too painful to design in prevention.” 2 of the cases I can recall were race conditions. In one case, the window was approximately 20 nsec, and the engineer said something to the effect “It’s so unlikely to happen, I’m not going to go to the trouble to prevent it.” I said to him, “If this firmware was in a product that your mother was using, and her life depended on it, would you have that same attitude?” Naturally, he decided it actually *was* worthwhile…

One other thing – you mentioned your early career as an electronics engineer & the worst case. About 3 years ago I was brought in to find & fix a very strange problem… customer thought it was firmware, in fact the hardware manager said (literally) “There is absolutely no possibility that this is a hardware problem.” Turns out it was hardware. Complicated circuit with many resistors, 10% tolerance parts used on the board, and guess what? The tolerances lined up in the circuit (almost +10% here, almost -10% there) such that readings & behavior were intermittently erratic. Original designer assumed a 470K resistor meant a 470K resistor, not possibly a 430K resistor.

Last thing – and I don’t mean to steal your thunder, since you’re going to be writing a series, but in addition to good design & implementation, I think process can have a huge beneficial impact on quality. I’m not a process fanboy, I’m in the “just enough to make sense” camp, but coding standards, code reviews, static analysis, version control, regression testing, etc. can all have huge impacts on the product’s shipping quality. Seems silly to even have to mention these activities, they should almost be a foregone conclusion, but sadly I can say that is *definitely* not the case.

I have certainly been neglecting my blog but have resolved to get back to it – though I don’t have so much time for it as I had a year or so ago.

I’ve been reviewing SKC++ and will revive it, with some changes, probably under a different name. I made the prototype; now I’m going for the product!

Will you be continuing with the SKC++ series? I enjoyed that & hope that you have time to resurrect it!

I got around the problem by allocating the initial pool space dynamically instead of declaring an array. (Static allocation, as originally attempted, would have been better, but I couldn’t figure out a reliable way to automate it.) The pool memory can be taken from the standard heap (if you must); the actual pool is never given back, so there is no fragmentation issue. The preferred method, though, is to use my super-simple allocate-only code for obtaining pools. This is a configuration option and it works completely transparently – the user doesn’t have to manage the initial allocation of pool space, except to ensure that sufficient total space is made available.

If you haven’t already downloaded the code, I suggest that you do so and have a look through it. I expect to be updating it again soon, and will post another message when I have done so.

My code, which you can download if you wish, can’t detect or prevent the coding errors which cause memory leaks but definitely can get rid of the fragmentation problem inherent in all heap (variable block size) allocation systems. No tool can help you with this! IT programmers can generally ignore fragmentation and get away with it; embedded programmers most certainly cannot.

Actually, I lied. The Managed class and its associated smart Ptr class are now part of my software bundle and proper use of these can help greatly to prevent memory leaks.

Thank you for your interest.

The three basic principles which guided me in the design of SKC++ were that it should be:

- Easy to learn
- Easy to use
- Difficult to misuse

My own experience and that of other engineers that I have asked about it is that we like to reserve our time and creativity for the challenge facing us. We don’t want to waste too much time learning how a tool works, or agonising about its options; that should not be part of the challenge. I regard SKC++ primarily as a tool for Application Engineers.

Richness does have its place and there is an argument that you appeal to a greater market if you provide all the traditional, “expected” features as well as any new ones. However, my mission these days is not to sell millions of copies of SKC++ (though more than one or two would be nice), but to distil decades of experience into what I do and deliver only the best bits, thereby making my small contribution to improving embedded software quality. (For the historical enlightenment of those starting out in the profession, I might occasionally blog about the less good bits, but that’s another subject.)

Of course the problem with Occam’s Razor is that you can slice off too much. But to counter that by adding redundant features “just in case” is cowardice. There’s a thin line somewhere in between and that’s why I threw the Semaphore argument open.

Here are some other arguments for the lean approach:

- Quicker to market (though I still haven’t got there)
- Less to go wrong
- Easier to add features, on demand, than to take them away after publication

Who knows, I might make more money adding redundant features for traditionalists (or others stuck in their ways by choice or by circumstance) than I make from the basic product!

Thanks for your comment about DO-178B. I’ve moved it to the Software Download page (link above), and have replied to it there.

Getting a product accepted is about the user experience. That applies as much to an RTOS as to a toy like an iPad. Have you ever stood behind someone while they use a program you are a hot shot at, say Excel, and suddenly notice them doing something in quite a different way to how you would? That’s richness.

The other argument in favour of richness is that it gives uses more opportunities to perhaps use the product in ways you, the maker, didn’t anticipate. Which of course in an RTOS may not always be a Good Thing

David Stonier-Gibson http://splatco.com

To those people:
You are still registered, of course, and I have changed nothing in your user accounts!

- Too many system calls
- Written in C