Modern testing for modern stacks

We have gotten into the habit of thinking deeper about one topic on a weekly basis. We pick topics based on anything interesting we read - so the topics can range from 'how to express the value of testing' to 'Dieter Rams' design principles' to 'effective remote work habits'. Employees are guided to spend no more than one hour researching the topic online. The emphasis is on coming up with their own ideas and interpretations. We then meet as a group to exchange ideas. I love this habit and consider it one of the more unique benefits you will enjoy at Qxf2.

Topic: Testing Internet of Things

20-Sep-2016

Avinash posted a very good interview about testing Internet of things. We really liked it. We think we should try and think about testing problems when dealing with the Internet of things.
References:
1. https://www.stickyminds.com/interview/developing-and-testing-games-internet-things-interview-jane-fraser
2. Pokemon Go issues: http://blog.smartbear.com/performance-monitoring/pokemon-go-software-quality-lessons/

Our thoughts

Arun

Some key things that stood out to me. One, the environment is part of testing. It's not as though you have a browser or a device as a platform that limits your user' behavior. Two, components are super cheap which means it is very easy for people cobble together products ... so the integration piece is vital. Three, development cycles are long (example of 3 days to figure out the sunlight bug). Four, thinking from a user's perspective is still the best approach to get started. Five, the users vary from adults to kids to even dogs :)

Avinash

I would like to be paid to play and test games. With IOT the way we interact with software and send inputs to the device or software itself would vary. We think of only inputs which we feed through hands like click, scroll etc but with IOT we need to think of how to simulate pulse, smell, mood etc which would act as inputs for IOT devices. Checking how Hardware works is going to be critical. Data transmission from device to the system also would be vital. Crowd sourced data for testing will play a vital role. Many smaller companies will come into fray. Bottom line would be less, testing budget would be less. Only solution is testing has to be smart. Lots of other domains will benifit so we may need to see how we can test devices related to Farming, Medical, consumer goods, games and lot of new things. Testing it remotely will be tough.

Annapoorani

Two years ago we heard the smart watch is in our comics book,a year ago it was so expensive to buy.But now we can buy it .The internet of things is quickly making the world relies on a drive.And it still needs to be tested.What should be considered while testing the internet of things? While software that runs on a browser can give hints and subtle clues about how to use it, IoT devices have no room for detail explanation. So instead of trying to understand how it works and what are all the functions are there we can try with minimalism is one thing that we can look for, more specifically, we can look for the things that are missing. And the second thing is set of hardware devices needed for testing,instead of buying all the devices to test,check what are all the devices are important to have,and which might be ok to leave out of testing rotation for now.The important thing to be considered is ,seeing what really happens when we really lose a connection.Always consider about security issues.

Smitha

This article was completely a new idea and learning to me. It's interesting & makes me want to read more about it. The future of testing is heading in a newer direction for sure. Environment is key for testing and we could pick the lessons as highlighted in the article.

Indira

The role of QA is critical in testing internet of things. The biggest challenge being replicating the environment.It is very expensive to replicate the environment required for IoT testing and demands too much of effort. Testing individual components is not much complicated but when it involves testing of subsystems, sub-components, and services that are interrelated are possessed by various groups and third party units it becomes a challenge. If user is unable to access a single dependent sub-component, it could affect the testing of the whole system. QA testing team also need to have sound understanding of the architecture, the Operating System, hardware, applications, protocols.Protecting user data and identifying, implementing security controls is the next biggest challenge.

Shiva

IOT testing is going to be a challenge. Integration testing is going to more important than ever. Ascertaining the device that malfunctions is going to be vital. APIs would form the base of any testing effort. Like mentioned in the Pokemon Go post - " An integrated GUI and API testing tool can help resolve the first issue. An ability to pinpoint issues at the GUI or the API layer allows GUI testers to cut down on intermediary steps" an interface testing tool would be useful to pinpoint the issue. Testes may also need to know the configuration details of devices involved. Testing is not going to be a trivial practice. It would also reiterate the idea that testers need to be a Jack of all trades.

Rohan

I like the interview of Jane Fraser. She spoke very well about general hardware testing. Anki is toy making company and it's good hear that, they throw's toys here and there during testing like kids throws toys while playing with them. Testing IOT is very critical and challenging. We can divide Testing in different areas like connectivity testing, security testing, compatibility testing, functionality testing, performance testing and exploratory testing. Out of these 6 areas, last three areas are common for all hardware testing. Connectivity testing is heart of IOT testing. Connectivity, Security and compatibility testing are related to each other. To test connectivity, tester needs to develop protocol simulator. Different devices use different protocol system that will be another challenge during compatibility testing. Once IOT devices connected to internet, the chances of hacking will be very high and now-a-days IOT devices are using for military purpose, bank security purpose. This makes security testing even critical. As a H/W tester, I always give equal priority for both unit and integration testing because many time I observed identified the problem is very critical after integrating the complete device and it's time consuming too. So I prefer to do unit testing before we integrate all units to make a product/device.

paper cut