Methodology and methods
for Part 1 of the study conducted December 2007 - March 2008
- 2 Methods
- 2.1 Pre-pilot methods
- 2.2 Pre-pilot results
- 2.3 Pilot methods
- 2.4 Pilot findings
- 2.5 Conclusions from the pre-pilot and pilot
- 3 Methods for the Part 1 main survey
- 3.1 Timescale
- 3.2 Sampling
- 3.3 Data Analysis
- 3.4 Response Rates
1 Methodology
For the study as a whole, the methodology is both qualitative and quantitative, with each approach used to inform and guide the other on an iterative basis, to help in refining the research instruments and in collecting data. Part 1 of the study, the 'environment map', began with a small scale qualitative pre-pilot in which SLT members were interviewed to identify issues to explore via a survey, to identify the most appropriate way of conducting the survey to be sent out in January 2008 and also, in preparation for the rest of the three year study, to identify appropriate incentives for schools to participate in all aspects of the study.
This section begins by describing the pre-pilot and subsequent pilot of draft research instruments and also gives the findings of the pre-pilot and pilot, since these impacted on the methods used for the survey that formed the main research method. The pilot of the questionnaires aimed to obtain feedback on format and content prior to the main survey.
2 Methods
2.1 Pre-Pilot methods
In December 2007 a pre-pilot interview schedule was developed for interviews with individuals responsible for the recruitment and retention of NQTs in 10 schools. Throughout December 2007 a total of 11 pre-pilot interviews were conducted in these schools, face-to-face or by telephone. Of the 10 schools involved, 4 were secondary and 7 primary. The majority of the schools were from Sheffield Hallam University's catchment area and included urban and rural schools, however, the sample also included schools from other areas/cities including London, York and Manchester.
The main areas explored in the pre-pilot interviews were:-- what questions should be asked in the questionnaires in relation to recruitment and retention issues;
- the most suitable format for the questionnaires i.e. online, paper-based;
- the most effective way of maximising response rates i.e. incentives;
- the most appropriate way of encouraging schools to participate in the 3 year longitudinal project as well as the 'environment mapping' survey.
The findings from the pre-pilot interviews informed the development of the questionnaires to be piloted, including questions to be explored about the recruitment and retention of NQTs, questionnaire format and incentives for schools to participate in the study as a whole.
2.2 Pre-Pilot results
Of the 11 interviews conducted in 10 schools, most were with Head-teachers (6). The remaining interviews were with Deputy Heads (3) and other staff responsible for NQTs (2). All those interviewed, apart from 2, were responsible for the recruitment of NQTs; the 2 not responsible for recruitment were either involved in the recruitment process or had responsibility for NQTs once recruited.
Recruitment
Pre-pilot interviewees were asked about the recruitment of NQTs, in particular about recruiting high quality NQTs. A wide range of issues were mentioned by primary and secondary schools, with little variation between them.
One of the most common problems experienced when recruiting NQTs is the quality of NQT applications. For example, one primary head-teacher commented that 'the standard of applications was disappointing because they lack flair' and a secondary head-teacher felt that 'the school had lots of applications for posts, but sometimes standard of applicants are not satisfactory'. In one primary and one secondary school, lack of suitable references was an issue. For example, the majority of applicants' references come from college tutors, and some head-teachers thought they did not know the student well enough and that references from the applicants' placement school would be more reliable.
Other issues related to the recruitment of NQTs are skills, experience and commitment. For example, one concern is that NQT applicants have limited experience in a variety of different teaching environments.
'I look for students who have had three effective teaching practices, who have done lots of whole-class teaching. The problem with many applicants is that they have not had enough time doing full-time teaching' (primary head-teacher)
A deputy head (secondary) saw the main concerns in recruiting NQTs as being that NQTs need excellent subject knowledge and a commitment to develop their own practice. One primary head-teacher looked for 'personality, commitment to the children, understanding of children's needs, curricular knowledge, team working skills and professional commitment'.
Retention
The pre-pilot interviews also considered the retention of NQTs. Respondents from primary and secondary schools felt that many NQTs leave after their first year because of lack of training and support.
'We do not have any problems retaining NQTs in this school. I think that it is because we make it challenging for them in a positive way, and we also allow them time for training and visits to other schools.' (primary head-teacher)
'Training has an impact on the retention of teachers. Schools have increasing responsibility in the process of teacher training, and have little knowledge of how to train teachers and in many cases little understanding of the skills that they use in a classroom and how to develop these.' (secondary head-teacher)
Some interviewees in primary and secondary schools felt that NQTs leave because the job is not what they expect and the workload is more than anticipated.
'Workload is the main issue for the retention of NQTs. Some come in and have barely survived the course never mind the teaching. Once they have started the role they often come in and leave to try something else. I think this is partly the course they undertook and the person as an individual not making the right decision.' (primary head-teacher)
Most primary school respondents felt that staff development meetings and continuing professional development (CPD) were crucial in retaining high quality NQTs. These issues were not raised in any of the pre-pilot secondary schools.
Other issues raised related to retaining high quality NQTs were their lack of experience with parents and lack of classroom management. For example, one primary head-teacher felt that 'sometimes trainee teachers have no experience of dealing with parents whilst on their placements and once they start as an NQT in a school they can't deal with the parents'. Another primary school respondent felt that 'overall when NQTs leave it's because of the school and lack of understanding of the job and how they fit into it. Also, some NQTs are unsure about how to manage parents and in some instances NQTs are over confident'.
Survey Format & Encouraging Response
The pre-pilot interviewees were asked about a survey format, ways of maximising response for the initial survey and the 3 year longitudinal study and about possible incentives to increase participation.
There was a mixed response about an appropriate way to distribute the survey i.e. paper-based via mail, online or both. The most favoured method was to have both an online and paper-based survey, giving respondents the option to use either. It became apparent that some schools may not have online access whilst others use the internet frequently. For example, one primary school respondent felt that a paper-based survey was much more appropriate for their school whereas in a large secondary school online was preferred. Only one pre-pilot school felt that a survey was a bad idea, whichever method was used.
The pre-pilot interviews also explored the best way to encourage cooperation from SLTs in the 4,000 survey schools.
Responses included:- a letter asking for cooperation prior to the survey going out;
- appealing to individuals' professionalism;
- emphasise how the project will benefit the school;
- distribute the letter and survey at less busy times;
- incentives;
- endorsement from school bodies.
- certificate/plaque for taking part;
- training;
- school equipment;
- online tutorials;
- gift vouchers
- gift for the staffroom;
- cash;
- prize draws.
Respondents considered that the aspects that would encourage schools to participate in the 3 year longitudinal study were similar to the above, with the main ones being cash to release staff and recognition. The majority of pre-pilot schools agreed to pilot the questionnaire and half agreed to be a case study school.
2.3 Pilot methods
Two draft questionnaires were developed, one for SLT members and one for NQTs, both of which took into account the findings of the pre-pilot interviews. In January 2008 the draft questionnaires were sent to all 10 schools involved in the pre-pilot interviews. The majority received an online version of the SLT questionnaire via email and 2 were sent hard copy versions. In addition, the schools were also asked to give questionnaires to any NQTs or early career teachers in their school, either hard copy or online. SLTs and NQTs were asked to complete the questionnaires and give feedback on style, ease of completion, content and format, to inform the development of the final questionnaires to be sent out in early February. In total 7 SLT and 7 NQT questionnaires were returned.
2.4 Pilot findings
The pilot indicated that the initial questionnaire for SLTs was fairly time consuming to fill in and took longer that the 20 minutes estimated. It was decided to reduce the questionnaire in length, reduce the number of open questions and re-order some questions, for example, the number of questions about vacancies was reduced. The original section on vacancies contained subject specific questions for primary schools, however primary school respondents indicated that NQTs are not generally recruited on this basis but on the basis of the key stage they are trained to teach or other generic skills. The original section on retention asked for specific numbers of NQTs in the school who had left the profession and for what reasons. Respondents found such specific information difficult to provide, so the question was adapted to ask for proportions instead of numbers.
Respondents to the NQT questionnaire reported that it was fairly straightforward to complete and did not take too much time. The analysis of the responses revealed no problematic questions. This questionnaire was adapted in line with the changes to the SLT questionnaire, to allow for comparisons at the analysis stage.
2.5 Conclusions from the pre-pilot and pilot
The pre-pilot interviews, pilot questionnaire and the literature review proved very useful in clarifying the survey methods for the 'environment map', in identifying issues relating to the recruitment and retention of NQTs and items to be included in the final questionnaires. In the original tender document it was proposed that the survey be online. However, as a result of the pre-pilot interviews it was decided to also use paper-based questionnaires. The pre-pilot gave an insight into what would be attractive incentives to schools to increase response rates and encourage further participation. Consideration was given to all the incentives suggested and the most appropriate in terms of time, money and feasibility were vouchers, cash and prize draws.
3 Methods for the Part 1 main survey
3.1 Timescale
The timescale was as follows:
- initial letter with publicity leaflet sent 7th Jan
- postal survey sent 7th Feb
- hard copy reminder 27th Feb
- 1st email reminder 3rd March
- 2nd email reminder 7th March
- 3rd email reminder 10th March (from TDA)
- telephone chasing 14th March
- survey closed 27th March.
3.2 Sampling
The aim was to survey 4,000 schools in order to achieve a response rate of 1,000. The sampling technique originally proposed was to take 30 of England's Local Authorities and survey all schools within them, providing a representative cross section of schools across the country. This was based upon an 'average' (mean) number of schools per Local Authority area.
It was decided that the sample would need to proportionately represent rural and urban areas and each of the nine English regions. A list of all 354 Local Authorities was prepared and broken down by the nine regions and by low, medium or high population density (using 2002 national statistics data ). In the table below N is the number of Local Authorities within each region and the proportions within each population density are given. The total number of Local Authorities sampled initially was 30.
Table 1 Number of local authorities and related population densities by region
(N = number of local authorities)
| High | Medium | Low | ||
|---|---|---|---|---|
| % | % | % | ||
| East | N=48 | 22.9 | 39.6 | 37.5 |
| East Midlands | N = 40 | 20.0 | 37.5 | 42.5 |
| London | N = 33 | 100 | ||
| North East | N = 23 | 26.1 | 47.8 | 26.1 |
| North West | N = 43 | 37.2 | 44.2 | 18.6 |
| South East | N = 67 | 31.3 | 44.8 | 23.9 |
| South West | N = 45 | 20.0 | 11.1 | 68.9 |
| West Midlands | N = 34 | 32.4 | 29.4 | 38.2 |
| Yorkshire & the Humber | N = 21 | 14.3 | 42.9 | 42.9 |
To produce the sample of Local Authorities, all 354 were listed alphabetically by region and assigned a number from 1 to 354. An online random number generator was used to select a sample of 30 Local Authorities, with any above the target number for each criteria (i.e. region and population density) discarded. A list of schools within these 30 Authorities was created using data from various sources, including the DfES, Edubase and Ofsted, however this list did not provide the 4,000 schools needed for the sample: many low population density areas have small numbers of schools and a small number of high population density areas tend to skew the figures. A further 28 Local Authorities were therefore added to the sample. In total the number of schools sampled was 4,098. Each school in the sample is identified by its URN (Unique Reference Number), a national identifier that allows the sample to be matched with other datasets (for example, to use data held on free school meal entitlements). An additional 7 Local Authorities were later added to the sample, as a number of schools were discarded as ineligible for the criteria of the study (targeting inappropriate age ranges, etc). This brought the final number of Local Authorities sampled to 65.
Sampling for telephone follow upThe mail-out and the postal and email reminders did not provide the hoped for response rate and it was decided to follow up some non-respondents from secondary schools by telephone to enquire why they had not completed the questionnaire and to encourage them to do so. This required a representative sample of the existing sample of schools. Firstly all the secondary schools were identified and allocated consecutive numbers. A random number generator was used to select 230 and those already responding to the survey were eliminated, leaving 182 secondary schools, verified as being statistically representative of the original sample, all of which were telephoned. Reasons for not responding were recorded.
3.3 Data Analysis
The paper questionnaires were electronically scanned and the resulting data were combined with those from online responses. Each dataset was cleaned using a standard procedure, namely checks to ensure that any routeing instructions had been followed, range checks on all appropriate variables and logic checks to highlight obvious inconsistencies in the data. In addition 10% of the questionnaires received a complete quality check. The datasets were then analysed using SPSS version 12.
Each questionnaire contained several open-ended questions. The responses were transcribed and manually coded. Frequencies of the coded comments were produced.
3.4 Response Rates
Questionnaires were sent to 4,098 schools and 538 paper copies and 168 online responses were received, a total of 706 questionnaires for analysis. A number of schools (113) gave reasons for not participating in the survey such as "no NQTs" or "no time". Including these as ‘non applicable’ gives a response rate of 18%. The sample restricted the proportion of independent schools surveyed to 9% of the total (i.e. 361 schools) and 34 responded.
All 4,098 schools were asked to give the NQT questionnaire to all their current NQTs. In total 272 NQT responses were received: 180 paper and 92 electronic.
Joanne Gledhill, Peggy Haughton, Jason Leman, Anna Stevens, Sue Drew, Colin McCaig 2008
Methods: Summary