R&D Handbook

R&D Project Checklist

This page is basically a set of checklists that I originally created for myself.
It is still work in progress!

Scope Document

It is surprising how differently people can visualize a project. Individuals in Upper management, Sales, Marketing, Project manager, Development Team etc, might all have different visions. They might not even be aware of this discrepancy. I recommend that the Project Manager and Steering Committee agrees on a scope document before starting up with a development team. The scope document could contain:

  • Deliverables
    What is the actual outcome? An executable-file?, new HW with new SW? old HW with new SW etc.
  • Definition of Done
    Is the R&D project also responsible for starting up Production? Market introduction? Training of sales-org?
  • Platform
    Do we expect to run a desktop program on Linux, Windows or Mac? If it’s embedded – do we already know that it will run on Linux, or something else?
    If it’s an app – is it IOS or Android – or both?
    If it’s a cloud-solution – are we already “married to” Google, Amazon or Microsoft? – or do we spend time on evaluating each of these? Will we phase the introduction on some of the above – e.g. Windows first, then Mac?
  • Geography
    Is it worldwide, regional or local? This can mean a lot in relation to compliance – and thereby the work that goes into fulfilling this.
  • Staffing
    Will the project run on existing resources? If so, someone needs to decide what to do less of. If not – then who?

Requirements Documentation

Even agile projects will typically have requirements. This can be a word-document, an excel-sheet or a database – like e.g. Azure-Devops, Jira or something really big like Jama.

Requirements must be:

  • Testable
  • Traceable (where does it come from)
    If your company loses a big order, or decides to pull out of a market, it is important to know which requirements are not needed anymore. If you decide to involve companies to help test the prototype, it makes sense to go with the ones that will see most of their requirements fulfilled.
    If you are ever in doubt about a requirement, you need to be able to find its origination and dig into it.
  • Prioritized
    Use e.g. “classes” – like “A” for mandatory, “B” for important, “C” for Nice-to-have and “Z” for not-needed. It is a huge advantage also to know what is NOT required.
  • Atomic
    It is bad practice to put several requirements into one. If a device needs to be operational between -20 and +70 Degrees Celcius, you should create a requirement for each of the two limits. In the end you may be forced to change e.g. the lower limit to -10 degrees. No need to go through everything that builds on the upper limit.

It is highly recommended to “test” the requirements before going too far into the project, using methods like e.g. QFD – also known as “House of Quality”.

The fastest design/implementation/test/release is the one you don’t need at all.

Design

Most projects may inherit existing designs that they build on top of, while a few starts all over. In both cases there are often changes to the design along the way. I recommend that design is described in e.g. Power-Point form with the team before it is implemented. These Powerpoints can be reused to update the design documentation before and/or after implementation.

The following “design views” are all important to me although I don’t use all the sub-diagrams, -patterns etc:

  • Presentations/Discussions and Documentation: UML Diagrams
    • Use-case-Diagram
    • Sequence-Diagram
    • Class-Diagram – Containment,(Inheritance)
    • Structure Diagrams
    • Deployment Diagram
    • Statemachine-Diagram
  • Parallel work: Scheduler/Kernel/RTOS/OS
    • Building Blocks: Tasks & Interrupts, Messages, Queues, Shared Memory
    • Parallel Processing: Reentrancy, Thread-Safety, Atomic operations, Mutexes, Critical sections, Priority (-inversion), Round-Robin
  • Breaking down work: Gang-of-4 Patterns
    • Layers
    • Creational: Singleton, Factory, Prototype, Builder
    • Structural: Façade, Proxy, Decorator, Composite, Adapter, Bridge
    • Behavioral: Observer (Publish/Subscribe), Iterator, State, Strategy, Chain of responsibility, Command, Interpreter, Mediator, Memento, Template, Visitor
  • Low-level
    • Data-Structures, Memory-layout, Memory-types, Compiler & Linker
  • Database: Entity-Relations, Physical/Logical, Integrity, Constraints
  • UX/UI: Real-World Match, Status Visibility, Consistency, Standards and more (this is guidelines more than a design view)

Any design must be evaluated against many Design Considerations.

Coding

We will not discuss languages & code-constructs here.

For quality control the team should have at least:

  • A basic code-guideline
  • Static code-tests (preferably as automated checkin-gates)
  • Minimal formal review – like Pull-Requests (often related to git workflows)
    • Formal Inspections is even better

Debugging

I will not go into details about debugging. It is however important to remember that there are many levels of debugging:

  • Low-level
    Single-stepping, breakpoints, tracing, memory-dumps etc
  • Medium-level
    Print statements that are always in the code – normally disabled, but can be run-time enabled within “domains” like “network”, “audio”, “SPI” etc and typically also more or less verbose. Typically enabling all these debugging printouts at once will drag a system down, but you should have the extra CPU-power to open up for selected parts.
  • High-Level
    Statistics – e.g. on mailboxes; how much queue, which messages in the queues? etc.

When we talk about debugging, most people think about single-stepping and breakpoints. This is great if you have an idea of where in your code to look. To get to this point you may need to start with an eagles view – hence the other debugging levels.

Unit and Integration Tests

When is it a unit-test and when is it an integration test?

The important thing is that the team performs tests on their code before they release it to the main branch (often called master). And that even more testing is used when blocks are integrated. In many cases you can use the same tests for both types.

Obviously the new code should be tested. I recommend that small “test-journals” are written. It is so little work to copy a screen dump from an IDE or a terminal to a Word-doc, but it can save you so much time later, when you start doubting what really worked and what you did or did not test. A test-journal is not a formal document as such, but do yourself a favor and keep it for a year.

“Smoke-tests” of the existing code can also be advantageous. The more these can be automated – the bigger the chance of them being run.

Regression Tests

The more people are involved in a project – the more important it is that nightly regression-tests are run. These will often catch unforeseen side-effects. If the project is e.g. PC-software it is possible to run many parallel tests, and therefore have a great code-coverage in the tests.

Embedded projects may depend on hard-to-get HW-prototypes and may have sensitive wiring, or expensive prototypes. Such setups are not easily parallelized. This means that in order to run many tests, each needs to be of short duration. In this case it may make sense to run the longer-running tests over weekends.

The faster a bug is found after being introduced – the faster it is to fix it.

Typically regression tests are run on the main branch. By testing nightly we get a chance to find a defect within 24 hours. But a defect may have disturbed many people just by entering the main branch. For this reason many teams have gated check-ins where fast tests are performed before the code enters the main-branch.

Another approach is to have a “semi-main” branch where code is checked in, and tested nightly. Only if this is passed is the code allowed to find its way into the main branch.

When regressions tests are not just pass/fail, but also qualitatively measure parameters like Idle Time (%), Power-usage (W), Dynamic Range (dB), Packet-loss (%), GPS-deviation (m) etc – then you can depict these measurements on the Y-axis on a graph with time (or builds) as the X-axis. This becomes a valuable trend-analysis, where “jumps” are often related to check-ins – on purpose or not.

This builds up to continuous integration – see my review of The Phoenix Project.

Other Tests

Production Tests

Many embedded software teams are taken by surprise when the company’s own production department requests tests. Such unexpected work can be very time-consuming. It makes a lot of sense to raise the question early:

Are we supposed to deliver production tests – and if so, are there any ways we can plan such tests that we can reuse tests in our R&D setup? 

Customer Premises test

A power-up self-test is often necessary, but more elaborate tests can also be relevant. This depends on your product. An example is expensive B2B products that may be hard to return for service and are hard to live without while at service. The producer and the customer has a shared interest in uncovering as much as possible on premises. It is often possible to ask the customer to run specific tests and return the results in mails.

In the case of consumer products, it is more and more common for the product to autogenerate logs that are transmitted to a cloud-server when convenient – if the customer has approved it.

Defects

Bugs/Defects can come from internally in the project and from the market. In both cases it is important to register the defects with selected key information:

  • Symptoms
    What is observed. NOT what do you think is wrong (put it in e.g. details)
  • How to recreate
  • Details
  • Who reported the defect
  • What characterizes the system
    Versions of SW and HW.
  • Who is responsible for the defect now
  • State
    Travels between e.g. New, Don’t fix, Works as designed, Fixed in code, Released and possibly more states.

Version Control

Today git is dominating, but other systems can also do the job. The important thing is that:

  • You have a version control system
  • Check-ins are “atomic” and related to a single story/feature or defect. This allows you to rollback if needed, with known consequences.
  • For a given release you know what changes it contains (“shopping-list”)

Compliance

Before the project is too far it is important to understand the compliance requirements, so that the relevant work can be planned into the project.

There are several types of compliance:

  • Usage-Domain
    Medical and Automotive are known for their many compliance requirements, but e.g. Sound-Level meters and other devices used for measuring also have their domain-specific compliance requirements.
  • Implementation-Domain
    If a product has a wireless connection, there are many compliance requirements. These are country specific – like the FCC-rules in United States. Approval in one EU-country is valid for all.
  • General
    If you create HW for the EU you need CE-marking. This covers a lot of specific measurements like e.g. EMC and ESD.
    There are also IP-ratings (Ingress-Protection) for dust and fluid penetration. These are generally not required, but is something you can certify a product against – and then use in your marketing (business-to-business).

Steering Committee Meetings

Most companies have Steering Committees that help the projects to stay on track. It is important to understand that often the same people are in many SCMs. To ease their work, and their ability to compare projects,  all projects should report in the same “template” – often Power-Point. This will contain slides on:

  • Status on Milestones
    Some of the best milestones are demos. Most people can understand a well presented demo, and it gives credibility to the other measures.
  • Other Progress Indicators
    A good friend used to show a block-diagram of the product with components and interfaces marked with Green, Yellow or Red symbols for their current status. This is very relevant when introducing new HW with embedded SW.
  • Risk Analysis
    The classic format is a table with a line per risk. In the line we have the textual description, an estimate on probability, an estimate on effects on the product launch if the “thing” happens and some kind of multiplication of the latter two. A risk could have “High” probability of happening (marked red), but a “Low” effect (marked green) ending as a yellow result.
  • ETA
    Estimated Time of Arrival. Typically based on a curve of e.g. “Story Points” left over time. This is called a “Burndown Chart”. When will the graph cross the time-axis?
  • Hours Spent
    If a project is going nowhere it could be because the developers are doing everything else. Keeping some kind of track of the effort going into a project makes a lot of sense. You may be surprised.
  • Percentage of regression-tests passed and code-coverage in regression tests
    All tests may be passing, but what if we only test 10% of the code?
    There should be a goal for code-coverage – typically 100% is not really doable.
  • Market fit
    Has the market changed since the project was started?
    Has competitors introduced “nasty” products?
    Is the overall economy dictating new habits?
    Has the product moved?