Apart from the many advantages of testing during design, perhaps the most important is to test implementation of new technologies for which an ecosystem does not yet exist. Read on to find out how
Testing during design helps in verification of sub-system level design, which permits fine-tuning to meet the system specification requirements. The designs that are tested during this stage are of individual parts of a final product, which means that they would be missing criticial functionality that is required to make the system operable. This makes it important to simulate the other sub-components which are not yet available.
Alternatively, one could test subsets of functionality of the larger system that may be working at a given point in time, but not necessarily with all the functionality of the end product. “In industries like automotive and aerospace, where the end product is a complex system of systems, the ability to test subsystems through hardware-in-the-loop testing is a critical method to shorten the overall development cycle,” explains Raviteja Chivukula, Technical Marketing Engineer, National Instruments.
How it is different from testing after design
Testing is done during the design process by the design engineers themselves. Here, their focus is to ensure that the electronics system design works in the real world just as it did on paper. They would typically be debugging the product to ensure that the intended functions are implemented one way or the other, and then optimise those features correctly.
On the other hand, testing after design is more focused on ensuring that the product has met the design specifications that were set by the customer.
Stephen Hire, general manager, Aeroflex Systems Pvt Ltd, says, “From an R&D perspective, testing during design entails two very different objectives for the two approaches. Testing during design is about enabling the engineer to make a design work, then debug it and finally optimise it. Testing after design is usually more concerned with proving that the design has met its design specifications.”
How to successfully apply this methodology
The main point to understand and keep in mind is that the device under test (DUT) here is still not a complete system as it lacks many features. This requires the testing to be done in such a way that only certain parts of the system are considered, failing which the test would also invariably fail.
“The most optimal way to bring out issues is by doing unit testing of all the modules which go into a larger system. When systems get integrated, there is always a possibility of one corner case being missed, which is not handled properly during system level testing,” explains Amit Sinha, CEO, Dexcel Electronics Designs.
From a functionality perspective, many things in the product might not be working. The goal is to plan testing to achieve measurement objectives when the system is not yet exhibiting high levels of functionality. A spokesman from Teledyne Lecroy adds, “One aspect of this challenge is to have a good idea of the system’s partitioning, whether in terms of subsystems or subsets of functionality that may involve multiple subsystems. Yet another aspect is to have an understanding of what the capabilities of the subsystem or subset should be, and devise a test to verify that functionality.”
This can be explained by looking at cellular base stations as an example. During the earlier stages of a new technology’s evolution, such as in the case of LTE-Advanced, there are no mobiles available that use the technology. “This requires design engineers to use test equipment as a test peer to help them make sure that the base station is correctly implementing LTE-A protocols and test features such as carrier aggregation,” explains Stephen Hire. As the design progresses, they will then use the test mobile to help them debug any issues and to help optimise critical performance elements such as scheduling algorithms and data throughput.
Once the design phase is completed and the prototype base station is ready, it will usually be handed over to a different group for conformance testing to ensure that the base station meets all of the relevant 3GPP standards—this latter stage can be considered as ‘design after testing.’ Another aspect of testing after design comes in whenever a new version of software is released. For most commercial products, there is usually a lot of regression testing to ensure that all the features from earlier versions of software still work correctly with the latest version.
This design methodology involves product design happening hand-in-hand with test benches for the product. “In the automotive and aerospace industries, it has been practised with varying degrees of success in the form of the V-diagram model. While the use of concurrent design and test processes such as the V-diagram is common and obvious in these industries, adoption of these practices is growing in other industries and for other types of devices as well. For example, in the semiconductor and consumer electronics industries, shorter product lifespans and increasing product complexity continually drive pressure to reduce product development time by adopting the ‘test while design’ strategy,” explains Chivukula.
Traditionally, the left side of the V-diagram has been viewed as an R&D or design task and the right as test and validation tasks, and these two phases of product development have been separated. Once the production control unit is manufactured, it is passed to the validation engineers with a set of requirements to complete the verification & validation (V&V) process, and then report back on the results. However, in the concurrent design and test approach, there are real-time testing tasks going on throughout the entire process. “In the design phase, engineers perform model-in-the-loop testing by applying stimulus profiles or test scripts to simulations of the system and analysing its response. Once an algorithm is ready for prototyping, engineers take the prototype controller to the test lab or field test site for evaluation after which it is deployed. Testing occurs throughout the process,” adds Chivukula.
T. Anand of Knewron also stresses on the importance of testing during design, and cites an example where it helped his firm to remedy a problem quite inexpensively. “In one MCU-based design, we found that RTC value was not synchronised; rather it was giving random timing whenever invoked. Majority of the system functionality was dependent on RTC sync and so was a critical issue. By logic everything should have worked fine, but it didn’t. We could get out from that issue after changing declarations of RTC-related variables and also changing their types. Additionally, we had to change compiler/linker settings where it was optimising unnecessarily and causing trouble. If we would have skipped that test (assuming that RTC code is common and can’t have any issue) we would have spent too much time in final phases during release testing, which is costlier than early detection and remedy.”
Advantages to the design process
As mentioned earlier, in many cases, it is not possible to complete a design without testing during the design phase—especially for new technology. Apart from that, the other major advantages are risk reduction and cutting costs. If a design has a problem that is only discovered after the prototype is almost complete, the costs of correcting it can be enormous. Stephen Hire explains, “If the problem is at the heart of the design, then a lot of rework will be required; and where companies are pushing to meet a specific launch date, a problem found late in the design phase can cause big delays. Or even worse, the problems may only be found after the launch when customers start complaining and returning products to the manufacturer—every manufacturer and designer needs to minimise the risk of this happening.”
Teledyne Lecroy spokesman gives us a different point of view, “Product development is fundamentally a risk mitigation exercise, and testing gives you confidence that the risks have been mitigated. If a given subsystem has been thoroughly tested and verified, it should not be the source of issues upon assembly and testing of the entire system. If all of the subsystems have been similarly verified, faults in the end product can typically be traced to issues with the interfaces between the subsystems. Particularly with larger systems, testing during the design phase helps to partition the problems to achieve faster time to market.”
In the design of any large, complex system, testing during design implies that you know where you are in the process. Knowing where you are as you design can inform what you do at the next step of the design process. In general, if implemented properly, testing during design reduces, and has the potential for eliminating, long loops of redesigning. As a result, the development process tends to be more linear and predictable and, consequently, the time to market is reduced.
In short, the advantages can be summarised as: (a) an ability to test leading-edge functionality where there is no alternative; (b) enhanced ability to optimise the design for superior performance; (c) a lower risk of design problems being discovered late, causing costly rework or delays; and (d) reduced risk of financial losses or reputational damage.
Most leading consumer and industrial electronics manufacturers already use this methodology, because they understand that the costs and risks of not doing so are too high. Also, if they want to be ahead of the competition, they cannot wait for technology to mature—so they need to test during design. However, for companies working on mature technologies, some regard the costs of investing in additional test equipment early in the design stage as too high. Stephen Hire agrees, “It is easy to quantify the costs of extra test equipment but many engineers forget to factor in the costs of product launch delays, excessive customer returns or damage to a company’s reputation from inadequately tested products. Let’s take an example of a product with a selling price of Rs 5000 and an expected monthly sales volume of 10,000 units. Every one week of delay for the product launch will cost the company approximately Rs 12,500,000 in lost sales revenue; and that doesn’t even account for the cost of faulty products returned or the damage to reputation. A shortcut in the design stage can quickly add up to a huge financial damage.”
A typical challenge of testing during design is of simulating the conditions of the overall system during lab conditions. “In order to simulate the right conditions, it typically calls for extra effort in writing test cases and test programs, which some people might consider as an overhead during design. And because of this, some engineers might refrain from doing so,” adds Amit Sinha.
In general, the time-to-market pressure is the biggest factor. There is also a lot of pressure to see the entire system working. Unenlightened management might see an involved process of thoroughly testing subsystems as a waste of time. According to Teledyne Lecroy spokesman, “There is often a conflict between testing at the system and subsystem levels, and achieving a balance between them takes much thought and planning. Designers who are trained primarily in design alone must think about the testing as much as the design. More experienced engineers do a better job of factoring in testing, because they have suffered the consequences of ignoring it. Engineering schools do not teach their students to approach design with testing in mind. Rather, it is learnt with time and experience.”