Sunday, December 30, 2012

Visualizing architectural dependencies based on custom metrics...


Using dependency models to manage complex software architecture

http://delivery.acm.org/10.1145/1100000/1094824/p167-sangal.pdf?ip=129.16.213.178&acc=ACTIVE%20SERVICE&CFID=211948687&CFTOKEN=15311002&__acm__=1353930983_bd262d8f7f2a015ed0637a3abfa321f8

Looking for a related work on my newest article I've encountered this nice paper describing how to visualize dependency structure matrices (DSMs). The associated metric is quite simple and can be automated using static code analysis tools. What is even more interesting is that one can use many metrics and use the same visualization.

There are a few highlights which I find useful:

  • The authors show a tool which can be used in industrial applications
  • The tool and the method were evaluated based on large open source code
  • The tool has the possibility of zoom-in for components to show inter- and intra- dependencies
I need to test the tool myself with a few metrics of my own. I'll come back later with the results. 

@MiroslawStaron

Sunday, December 2, 2012

Fault injection and mutation as a means of ISO 26262 vV&V

Injecting faults and mutating programs  - towards a uniform method?
http://ieeexplore.ieee.org/ielx5/5260979/5260980/05261017.pdf?tp=&arnumber=5261017&isnumber=5260980

This paper shows a method for modeling fault injection environments and mutations in a uniform way. When modeling mutations, the method seems to support even the execution.

What I like about this paper is the fact that they try to use the same method for both injections and mutations. What is still to be done is the validation. Of this approach in an industrial context. Let us wait for that...


@MiroslawStaron

Monday, November 26, 2012

Fault injection - overview of techniques...

Fault injection - a vital technology for evaluation of safety critical systems.
http://dx.doi.org/10.1109/2.585157

This paper presents a nice overview of what kind of techniques for fault injection exist and where to look for more information on them. It also lists example tools for each class of the technique that the paper discusses.

In ISO 26262 this fault injection is important as it allows to test the safety mechanisms themselves. Even though in theory this is a simple task, in reality it is far for trivial to observe how the safety mechanisms, e.g. sandboxes, work when something gets faulty. One could do a lot of testing, but the most important part is to see whether the runtime mechanisms handle the situations in software that could cause hazards. Fault injection is also required for higher ASIL levels I chapter 6 of the standard.

@MiroslawStaron

Agile and Lean in Finland

How much agile is there in Finland?
http://dl.acm.org/citation.cfm?id=2372275  

This paper presents an interesting survey on how the Finnish industry adopts Agile and Lean software development principles. 
The paper shows that the majority of companies follow Agile principles (33%) and Agile+Lean (21%). What the survey also showed is that Lean principles on their own are not that common and are used only by 2%of the surveyed companies. 

Out of all the Agile methods it is Scrum which is the most popular, while TDD is used only by 1% of companies adopting Agile. 

This paper is a great view on the current status of adoption in industry (perhaps not all of it though). It gets even more interesting if we can consider this paper together with the work of Korhonen (see my previous post at: http://semetrics.blogspot.se/2012/09/supporting-agile-transformation-with.html).


@MiroslawStaron

Sunday, November 25, 2012

Great examples of Lean QA

Cases of QA in Lean software development...
http://lnkd.in/SYcD_F 

This weekend my Twitter feed showed a series of great articles from Tom Gilb @TomGilb where he provided a set of really interesting examples of companies who moved towards Lean.

The examples include Google and how they work with Chrome. What I like about that example was the analysis of short- and long- term costs of low quality. Not monetary or resource costs, but branding costs, for example - variety of programming languages which make it hard to debug lead to winning grounds of languages like Python or Ruby.

Another case from Hanssen which he shows is a great example of how companies engage customers and shorten interations.

Following the link above opens up a lot of great material about QA.

@MiroslawStaron

Wednesday, November 21, 2012

More projects == better defect predictions?


More efficient defect prediction when using data from multiple projects?
http://dx.doi.org/10.1016/j.infsof.2012.10.003

By looking at the newest research in the defect prediction field I've discovered this piece of work which intrigued me a bit. Usually we build statistical models in forms of equations describing defect inflows or use analogy based estimates - we use historical data to create models for new projects. This usually works fine, but this paper discusses things one step further, namely (and I quote):

RQ2: How much within project data should be enriched with data from other projects to achieve comparable performance with full within project data predictions?


The results show that using only 10% of the data can yield results of the same quality, which can significantly improve the cost-efficiency of defect predictions in industrial contexts.

@Miroslaw Staron

Friday, September 14, 2012

Supporting Agile transformation with defect management...

Supporting Agile transformation...
http://dspace.cc.tut.fi/dpub/handle/123456789/21063

Usually I write about articles, but this time I wanted to point out a doctoral thesis work from a large telecom company in Finland. The company has undergone a transformation from project-driven to Agile software development and it seems that they are gaining on the market, looking at more feature-rich products and faster cycles.

I especially recommend Part IV where they discuss the effects of Agile transformation on defect management and quality. Not many companies were able to measure such a strong effect. I will use this reference often to point toward numbers :)

@MiroslawStaron

Friday, August 10, 2012

Relationships about commenting the code and software bugs...

On the relationship between comment update practices and Software Bugs - a recent paper from JSS
http://dx.doi.org/10.1016/j.jss.2011.09.019

Looking closer and closer into the field of Mining Software Repositories and finding dependencies between code and defects, I've come across this paper. It shows how insufficient or wrong commenting influences quality.

I am sure I need not convince anyone about the direction of this relationship :) However, i wanted to highlight the actual numbers that they've worked with:
  • In FreeBSD, there are 2x more inconsistency (i.e. misleading) comments that the consistent ones - this causes the developers to mistrust comments and skip them at all. A vicious circle
  • Eclipse has a majority of consistent comments - probably caused by the involvment of more rigid processes.
  • A link to the graphs showing these dependencies: http://www.sciencedirect.com/science/article/pii/S016412121100238X#gr2 
The results are interesting since they do not identify a linear dependency between consistent and inconsistent commenting and bugs. Instead they identify relationships like this:
- If the number of inconsistent changes are large, then adding new changes (even if they are consistent) usually leads to bugs.

To sum up - interesting article to read for all of us who want to build automated prediction systems for defects.

@MiroslawStaron

Wednesday, July 4, 2012

ISO 26262 and Agile practices

ISO 26262 and Agile practices:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/invisiblethread/entry/five_benefits_of_agile_practices?lang=en

This is a very interesting article tackling a very tangible problem in industry - footprint of standard implementation. The article explains how the process footprint can be decreased by adopting certain Agile practices.

The article shows that there can be possibilities of tailoring agile to heavier processes where (for example) safety requirements are not something to play around with.

In non-Agile processes, the standard compliance usually takes a lot of effort and requires a lot of documentation. This article shows how this can be combined with agile practices on "minimum" documentation.

Recommended reading for all interested in standards implementation - not only in the automotive industry.

@MiroslawStaron

Tuesday, July 3, 2012

Agile metrics from IBM

Software Econometrics
http://public.dhe.ibm.com/common/ssi/ecm/en/ral14037usen/RAL14037USEN.PDF

This is a nice paper that conceptually describes goals with metrics in the Agile software development. A very good starter reading. I like the concepts and the ideology, but what I lack in the paper is the depth of how to measure things.

Some aspects that come to my mind as and old "metric" guy are:
- How to define "change" in the cost of change metric? What is the difference between rework and change?
- How to define scrap and rework? How to measure those concepts in software development? Is the first feature that one develops to show proof-of-concept a waste - or is it a learning activity?

In short - nice article, but requires digging deeper into the context.

@MiroslawStaron

Tuesday, June 5, 2012

Selecting self-managing teams...

Selecting and understanding self-managing teams...
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=5287005&contentType=Journals+%26+Magazines&sortType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A5286993%29

The dream about self-managing, high performance teams is still out there. Although it seems that benefits of those teams are plenty, one could always start thinking about the environment where they function best.

In this article in IEEE Software the authors look at these environments. Shared leadership, common decision-making and trust seem to be one of the most important internal characteristics of the teams. Knowledge sharing and organizational support were the most important factors at the organizational level.

I recommend this article as it can open eyes on the fact that having x-functional teams (testers, designers, analysts,...) might not be enough to achieve high team performance. And high team performance is one of the first steps towards high organizational performance.

@MiroslawStaron

Sunday, June 3, 2012

Aligning decisions around RE - case study

ALIGNING SOFTWARE PROJECT DECISIONS: A CASE STUDY http://dx.doi.org/10.1142/S0218194006003002

 Looking for interesting insights on how decisions are made in software development today, I've come across this article which looks at aligning decisions around RE.

The article has a nice touch on the practice since it is a case study. The authors studied 5 strategic decisions, 10 tactical ones and several operational ones using a framework developed by them.

What I liked best about this article is the fact how the authors reported on the interplay between different stakeholders when discussing decisions. Since that report was based on the interviews, reading it allows one to reason how much the decisions were related to personalities of the decision makers.

Generally, very insightful for us who try to understand why politics in software development plays such a role...

@MiroslawStaron

Monday, May 28, 2012

Making decisions in Agile teams - another case study

Challenges of shared decision-making: A multiple case study of agile software development
http://www.sciencedirect.com/science/article/pii/S0950584911002308

I've came accross this paper with a lot of insights how agile teams reason when making decisions. This papers goes into details explaining the multiple quotes.

Here is an example of a quote from that paper:  It is now more difficult to ‘‘steal’’ resources from us, because the consequences of losing resources during a sprint are more visible with Scrum. Earlier the deadline was 6–12 months ahead, and then it was easy to steal a day or two. There has been a change of attitude in the company, and it is now well accepted that you do not steal resources from a Scrum team during a sprint.

Resource allocation is usually an issue in software projects and using Agile has this advantage that the allocation is onto the teams (to a large extent). This quotation shows also how they think when making the resource allocation visible to the management - e.g. through shorten deadlines. As you can see, this influenced the company.

@MiroslawStaron

Sunday, May 27, 2012

Simulating Agile teams...

Making decision about process changes based on simulating Agile processes...
http://www.springerlink.com/content/x8566274232m672g/

I listen to the talk presenting this research and, quite honestly - I expected something different. However, I could not stop thinking about this approach. Using agents they were able to recreate the behaviors of Agile teams based on real data from Microsoft.

I need to try out this tool so see how one could simulate process changes. Imagine if this could help out in making decisions on how to put together a team...

@MiroslawStaron

Sunday, May 20, 2012

Does Measuring Code Change Improve Fault Prediction?

Does Measuring Code Change Improve Fault Prediction?
http://dl.acm.org/citation.cfm?id=2020392

Code churn measures seem to be very nice predictors of software failures. In this paper, Bell, Ostrand and Weyuker show that one does not need much more. They also investigate whether code churning is often linked to other anomalies based on 18 releases of a large software system.

I recommend this paper for two reasons - it shows how simple measures can give a lot of value and because it comes from top names!

@MiroslawStaron

Tuesday, May 8, 2012

Four ways to speed up product development...

Four ways to speed product development (written in '94, valid now for SE)
http://dx.doi.org/10.1016/0024-6301(94)90209-7

When looking for solid research on how quality improvement impacts lead time, I've came across this paper. It shows that the product development models of the 90's are no longer valid. Instead, new aspects are important, like:
- streamlining development,
- parallel feature development, or
- continuous releases

Kind of obvious and already known in the 90's. How come software engineering discusses those things now?

@MiroslawStaron

Motivations (systematic review)...

Models of motivation in software engineering
http://dx.doi.org/10.1016/j.infsof.2008.05.009

I've written about motivations a few times, so let me write another paper review. This systematic review is a very good set of references for research on motivation in software engineering.

This paper proposes a model where they identify intristic and extristic factors. Again, technology challenges are one of the main drivers. What are the others then? Well, some are:
- variety of task
- feedback
- intellectual stimulation
- individualized consideration
- ...

The interesting thing is that they also put some numbers on how important different aspects are in a number of different models which they investigate.

I also recommend reading http://www.jstor.org/stable/249347 where the motivation is evaluated from the perspective of job satisfaction. This particular paper is a bit old, but still interesting.

Wednesday, May 2, 2012

Do social interaction in projects influence software quality?

Studying the impact of social interactions on software quality (paper review)http://www.springerlink.com/content/j747jqn77h358824/

I wish this paper could end up at some coffee tables in the lunchrooms at software development companies. I also wish this study to be replicated in large, commerial software systems.

In its essence the idea is simple (yet cool!): if there exist predictors of quality in the internal product characteristics (e.g. McCabe complexity), are there any predictors based on organizational aspects like social interactions.

This paper shows that in projects like Eclipse (www.eclipse.org) these social interactions, extracted from defect tracking discussions, have a good predictive power. The paper uses such aspects of social interactions as role or participant - it also includes lengths of discussions, centrality of the subjects, etc. 

There are a number of cool conclusions and metrics, but one I liked most: "Overall, we observe a strong ef fect of discussion f low inconsistencies on the failure proneness of f iles associated with the discussion." Basically it means that if the discussion is unfocused, then the module is fault prone - nice that they've managed to find a metric for that!

@MiroslawStaron

Monday, April 30, 2012

Why can't we predict software defects...

A Critique of Software Defect Prediction Models (paper review)...
https://www.eecs.qmul.ac.uk/~norman/papers/defects_prediction_preprint105579.pdf

In this classical paper Fenton et al. presents a number of good points why predicting defects is so difficult. Although a lengthy paper, it constitutes a great review of different attempts of contemporary defect prediction (contemporary in 1999).

A very solid which will be of grab use for researchers and practitioners interested in predicting defects.

@MiroslawStaron

Sunday, April 29, 2012

Visual analytics - survey at Microsoft (paper review)

Visual analytics...
http://thomas-zimmermann.com/publications/files/buse-icse-2012.pdf

I've come across this paper by accident by looking at one of my favorite places - Microsoft Research. This paper shows what developers and managers expect to have from visual analytics. Examples of aspects are:
- targeting testing,
- triggering refactoring
- release planning
- targeting training
- ...

Interesting take on metrics - and a nice piece of empirical research

@MiroslawStaron

Saturday, April 14, 2012

Guiding your test based on faults ...

Fault-based test suite prioritization for specification-based testing
http://dx.doi.org/10.1016/j.infsof.2011.09.005

The question which tests one should prioritize is an important one. Companies usually use test coverage as the guiding metric - generally the higher the average, the better. Smarter companies use metrics like test failure - when the test fails often it is executed more often.

This paper, however, shows a different approach. This technique bases test prioritization on how well the test cases detects faults.

Interesting work, which I will try myself in a near future...

@MiroslawStaron

Thursday, April 12, 2012

Which modules are risky...

How do you know which modules/components/subsystems are risky?
http://www.springerlink.com/content/r60258615x8l1877/fulltext.pdf

The question from the title bothers many test leaders, project managers and quality santas. The reason is that this information varies over time. Once complex modules can be of high quality over time if no changes are made... simple modules can decay over time and features can make modules grow in strange ways.

This is an interesting paper, but takes a while to "digest"

@MiroslawStaron

Making effective decisions in Agile software teams - what's the problem?

Obstacles to decision making in Agile software development teams
http://dx.doi.org/10.1016/j.jss.2012.01.058

I often look at metrics research and discuss the connections between metrics and decisions. In the "old days" managers were concerned with these issues, but now it seems that with the Agile world the decision-metrics are often a part of teams' work.

This paper discussed what kind of decisions teams make and what the obstacles for making the decisions effectively are. For example, one of the obstacles is the fact that collaborative decision making prevents experts from having their voice heard...

Recommended reading for those who often wonder why some metrics are better for decisions than others.

@MiroslawStaron

Monday, April 2, 2012

How to put together a great Agile team...

How to Balance the Size and Skills of Your Agile Team
http://my.gartner.com/resources/220700/220700/how_to_balance_the_size_and__220700.pdf?li=1

Gartner did a great job in showing typical skills of Agile software development teams. They've looked at a number of disciplines and drew conclusions that, e.g.:
Master (not automatically ScrumMaster):
  • Experience: Five to eight years
  • Productivity: Very high, 18 to 25 FPs per staff month

  • They also have tables that show how the teams are usually composed and what it might mean.

    Interesting reading

    @MiroslawStaron

    Saturday, March 10, 2012

    Measuring agility...


    Enhancing enterprise agility by deploying agile drivers, capabilities and providers
    http://dx.doi.org/10.1016/j.ins.2011.04.034

    I often wonder whether why we talk so much about Agility without actually asking ourselves - do we really need Agile? If we take cost overruns as one of the drivers, then we can quickly come to the conclusion that in Agile software development overruns are less frequent. Hmmm... but in Agile we replan and rescope very often, so there is no plan "per se"...

    Either way. I think that this is a nice article which discussed the problems of adopting Agile in large enterprises. It discusses some of the "why's" and looks at how the "why's" can be looked at.

    In particular I like Table 2, where they link the drivers (why's) with the capabilities of the organization and providers for the capabilities - e.g. trust enables flexibility which is required by the driver "respond to customer needs"...

    @MiroslawStaron

    Saturday, February 25, 2012

    Motivation in software engineering...


    Motivation in Software Engineering (review of systematic review)
    http://dx.doi.org/10.1016/j.infsof.2007.09.004

    I've read this paper and recommend it to managers who instantly seek how to motivate their employees. What I find particularly interesting is that they actually reflect on the question "Who are software engineers?" - please take a look at their research questions, they really do a nice work!


    RQ1: What are the characteristics of Software Engineers?
    RQ2: What (de)motivates Software Engineers to be more (less) productive?
    RQ3: What are the external signs or outcomes of (de)motivated Software Engineers?
    RQ4: What aspects of Software Engineering (de)motivate Software Engineers?
    RQ5: What models of motivation exist in Software Engineering?


    In the light of this I like this main result best: "M. 17 Identify with the task (clear goals, personal interest, know purpose of task, how it fits in with whole, job satisfaction; producing identifiable piece of quality work)" which is the number one motivational factor for software engineering professionals. This means that code-ownership practices stimulate and help the practitioners to feel needed. So, let me ask a question: How about off-shoring? 24/7 development? Who owns the code? Who is motivated?

    Ok, it was more than one question, but still. I've seen this happening in many companies - off-shore does not mean better.

    @MiroslawStaron




    When to stop testing...

    Quantifying software validation...
    http://dx.doi.org/10.1109/52.28120

    The article is probably the predecessor of modern statistical methods in testing. It postulates that statistics could help in finding out when to release. I would also like to point the attention that this article came out of Bell Labs, where maturity modelling - a way of statistical software quality modelling - has been practiced since the 80's.

    I recommend this article as a starting point and a reflection point for the testing teams... can we build simple models that would show that we've reached the saturation point? What if we relate it to the previous blog entry - coverate metrics.... hmm... I'll try that in my next development project.

    If anyone has more articles in this line - please do not hesitate to recommend them to me and I'll make sure to put them in the blog.

    Saturday, February 18, 2012

    Test Coverage and Post-Verification Defects: A Multiple Case Study

    Why your test coverage is never going to be 100%...
    http://dx.doi.org/10.1109/ESEM.2009.5315981

    Yet again MS Research has proven to provide interesting and nice paper. In this article they describe dependencies between test effort, test coverage and post-release defect inflow.

    It is interesting to note that the effort grows exponentially with test coverage while the post-release quality only linearly. I will recommend that to my students who often ask about the cost of quality....

    Looking forward to more research from MS.