Loughborough University
Leicestershire, UK
LE11 3TU
+44 (0)1509 222222
Loughborough University

Centre for Innovative and Collaborative Construction Engineering


Dr Murray Sinclair


The Development of a Tool to Predict Team Performance

Project Title

The Development of a Tool to Predict Team Performance


BAE Systems


C Siemienoiuch
R Haslam

Mr Laird Evans
Mr Steve Harmer

Director of Research:
Professor Dino Bouchlaghem

Research Period

2006 - 2010

The development of a tool to predict team performance

This proposal is made in the context of the NECTISE research programme. NECTISE addresses challenges associated with Through-Life Capability Management (TLCM) in the context ofNetwork-Enabled Capability (NEC). These two concepts (TLCM and NEC) represent the most significant transfonnations to take place in UK defence provision for generations and both entail significant cultural and technological changes. Both concepts aspire to achieve greater agility: for NEC it is greater agility in terms of military effectiveness; for TLCM it is greater agility in the defence supply chain. Capability delivery will rely on successful assembly, deployment, reconfiguration, and disbanding of teams. Within the supply chain teams that draw membership from many disparate organisations (e.g. Military, defence primes, tier 2 companies, ... ) will be need to be rapidly assembled to provide capability, which may include both building and supporting that capability. For NEC, ad hoc team formation will be fundamental to the achievement of the transitional and mature epochs. Prediction of team success and the principles for effective team assembly is, then, of fundamental importance to the challenges ofTLCM and NEC. The purpose of this research is to explore and develop a serendipitous discovery in another Ph.D. studentship that concerned assessment of team structures for given tasks and the effect of moving individuals of different abilities around the team structure. The discovery of a quantitative method for predicting team reliability in executing a process/procedure that is sensitive to the structure of the team may allow answers to be obtained to the following, related, organisational questions.

What might be the human performance costs of a change to a given operational capability? An example might be a software upgrade that potentially could remove an operator from a weapons team by reducing and redistributing human tasks. Another example might be the instantiation of a communications application for a distributed design team.

What is the most resilient team structure to carry out this duty? An example might be the preparation and firing of a new anti-armour weapon for foot soldiers in a range of scenarios?

If I create this team to do this task before this deadline, how likely is it that they will succeed? An example might be a rework requirement for a team in the design process.

What is the likelihood of success if the team decision-making process is impaired? An example might be a coalition team (i.e. different understanding, communication problems) working with impaired information (e.g. late, incomplete, partial, errors).

What is the estimated cost of using this team to achieve that goal? An example might be the estimation of costs for a contract bid.

These questions reflect the possible range of application of this new approach to the estimation of human­ team perfonnance. This range covers small-team, fixed-procedure tasks performed in different ambient conditions through to large-team, partially-ordered processes within a changing organisational environment. The approach has been developed so far in the context of the small team, so its extension to the first two of the questions above should be less problematic than the further extension to the last two questions.

Research focus

It is proposed to focus initially on the first question above, the 'change question'. It is well-known that the introduction of new techniques can cause a short-term, 'initial dip', in performance for trained crews, despite retraining, and later there can be unexpected regression to previously-learnt behaviour, under various conditions of stress. The research would explore the effects on performance of the team of the interactions between team structure, experience, training, and the operational environment in which the upgrade is in effect. These are obvious variables likely to be important in any scenario, but they are not the only ones of significance; other prime candidates can easily be envisaged; culture, situation awareness, and many others spring to mind. For this research to be useful to BAE Systems, it will be necessary to create a model of these interactions, and embed this within a useful, usable technique that engineers will pe happy to use- i.e. to create a tool that for perceived little effort will deliver relatively significant results. Hence, there are two primary aims for the project:

1. Produce a tested model/models that enable the calculation of quantitative answers to the question deemed most important by BAE Systems; initially, it is expected to be the 'change' question.

2. Deliver a technique that Design Engineers find useful to address this question.

Secondary aims can also be identified, given a satisfactory outcome to these two primary aims:

3. Explore extension of the model and techniques to answer the remaining questions listed at the top of this document.

4. Explore extension of the model and techniques to different groups of users; e.g. project managers, capability managers, and IPTs

Outline of research focus

The source of the research is in the domain of Human Reliability Assessment. One of the missing tools for engineers to use during the design process is one for assessing the probability of a team collectively making an error; all the tools and techniques to date assume it is an individual who makes the mistake. A reliance on the assessment of individuals is not tenable in team situations. Accordingly, a Ph.D. with Isabel Smith as the student has been sponsored by BAE Systems, completing next year, to explore the provision of such a tool.

The proposed research is to develop a basic model and associated algorithm with the intention of providing answers to the questions above (and others that might be suggested as the work progresses). It is proposed to do this incrementally, initially considering small teams executing well-structured tasks, and then extending the capability of the technique to larger, distributed teams carrying out semi-structured processes.

The initial users envisaged for this capability will be design engineers; if the capability meets their needs and uses, and can be extended, the next groups of users might be project managers and capability managers.

Human Issues Cross-Cutting Theme in NECTISE

The research outlined above links comprehensively to a number of the research streams within the NECTISE project. For instance, the Decision Support topic contains work elements associated with collaborative environments, and particularly the cost of collaboration; the Through-Life Systems Management (TLSM) topic will include a detailed study of the performance of IPTs and analysis of how these will, or will not, support the objectives ofTLCM. It is intended that the researcher (Murray Sinclair) would lead the human issues cross-cutting theme within NECTISE. It is expected that his forty years experience in the field of human factors will provide the expertise to identify and develop the key issues that are the subject of the theme and his long-standing relationship with BAE Systems will provide the understanding necessary to fully exploit the outputs of the theme within the Company.

The research proposed herein addresses some of the most fundamental aspects of the NECTISE aims; thus it will provide an enduring work stream to expose and mould the human issues that emerge within the NECTISE topics to ensure that important outcomes and learning are properly integrated across the programme.

Research Approach

The immediate research stages for investigation are outlined below.

• Develop a minimal but exhaustive set ofteam architectures, as a taxonomic tree. 'Architectures' here will include knowledge distributions, and the architectures created by the allocation of people to processes. Issues arising from complexity theory will obviously be significant in this effort.

• Identify a set of typical scenarios for systems change that are of relevance to BAE Systems. Given the MoD's interest in TLCM/TLSM within NECTISE, a likely topic for scenarios will be the effect of systems upgrades, or system impairments.

• Develop and validate the set of parameters that capture these architectures to model the team interactions in the scenarios deemed of interest.

• Explore in step-wise fashion the relationship between the numbers of variables needed for team interactions (currently three), accuracy and reliability, and the utility of the model, initially from a design engineer's point of view. The purpose here is to include as many of the variables as are deemed important to achieve a given level of accuracy and reliability, while avoiding the creation of an unwieldy, effort-demanding model that cannot easily be applied by these engineers, who have many other calls upon their time.

• Test the sensitivity of the emerging model and plausibility of the results in simulations, evaluated by SMEs. At this early point in the research, it is envisaged that a multi-agent model will be developed, featuring cellular automata, to be evaluated by inspection and by ensemble simulation.

• Develop a technique embodying the model that could be used within a design engineering environment. A naval example might be where a design engineer is contemplating how a new sensor module on a helicopter might be maintained on a heaving deck in the dark using the standard team, and wishes to know the likely failure rate.

• Explore the reliability, utility and usability of the model and the technique for design engineers in real, small-team scenarios initially in the military domain.

• Extend the model into large-team scenarios and other domains of interest.

Proposed research strategy

The motivation for this research strategy stems from a famous comment by Kurt Lewin: "There's nothing so practical as a good theory." It is the obverse of this that underpins the research plan below: "For a theory to be good, it must work in practice". Hence, the research will be based on industrial scenarios and will be strongly influenced by potential users. This is in accordance with the Eng.D. ethos.

To ensure that useful outcomes for the sponsor emerge as early as possible, the focus will be initially on small teams, within a given small set of scenarios of interest to the sponsor. It is anticipated that the research questions and outcomes will also feed into the major demonstration activities led by the NECTISE core research group.

Data-gathering will be based on standard ethnographic methods, with the analysis using standard statistical methods, supplemented by SNA techniques. At present, it is expected that modelling will be undertaken within MATLAB, The main sources of data for this are likely to be interviews with stakeholders, and analysis of the Demonstrators that are planned within NECTISE.



Contact us

+44 (0)1509 222623

The Centre Administrator
Loughborough University
LE11 3TU

Join us on Linkedin