Surrey Schools ICT Board - Broadband

12th September 2011

Draft by MW, then DB - Sunday 11th 2pm

Purpose of this presentation

• To provide an overview of the status of the broadband transition project to schools:

– What have we accomplished, and what are the obstacles we have overcome

– Where are we now & September Schedule

– Completing transition by 30th September

– Post-transition planning and decision process to complete the project - ICT Board meeting mid-Oct

– Support

What have we accomplished• School representatives worked with Surrey to devise

specification• Successfully took through tender and Schools

Forum process - difficult balance between performance and affordability across wide range of schools

• Over 95% of Surrey Schools are in contract

Obstacles

Challenges of public procurement process (OJEU) - see legal opinions re proposals re NHS and risks

Michael Gove removed half of capital funding just before contract was due for signature in July

Legal challenge from unsuccessful bidder

Consequences of delay in signing contract

• Easynet have charged for extending existing contract (which is at higher revenue rates than new RM) and there would be extremely high cost to going beyond 30th Sept

• This has led to compressed transition period and very tight implementation schedule, making us vulnerable to delays from other sources

It’s been like a marathon, and we are going through the “wall” at the 20th

mile before the last few miles!!

What have we accomplished - exchanges and core network

• Core distribution ring complete

• Formally tested 1st Sept and accepted

Farnham

Camberley

Godalming

Woking

Guildford

Thames Ditton

Leatherhead

Walton-on-Thames

Bagshot

Deepcut

Ewell

Burgh Heath

Merstham

Aldershot

Bramley

Clandon

Bookham

Dorking

ReigateOxted Upper

Warlingham

Redhill

Chertsey

StainesAshford London

Weybridge

Brookwood

ByfleetOttershaw

1GB links to Telehouse from

Walton-on-Thames and Leatherhead

Core network and testing

• So all 29 exchanges linked to Telehouse via Leatherhead and Walton-on-Thames 1GB backhauls

• Backhauls and distribution network testing completed successfully on 1/9

• The maximum outage seen was 25 seconds during the testing of simulated failure of the Walton on Thames and Leatherhead backhauls

• Testing of a simulated failure of the Core backhaul between the Farnham and Godalming exchanges resulted in zero outage as traffic immediately rerouted round the distribution network.

BT Openreach

• Problems caused by stipulation that information about capacity / excess dig costs etc is only given when order is placed (with associated lead times)

• And an Easynet “cease” has to give 90 days so has to be placed before you know from BT if there is going to be a problem

• Easynet have not shown any flexibility

• Delays and changes to the core design caused by insufficient capacity in BT exchanges for “unbundling” per se without long delays for extra work (too long for the project and 30th Sept date)

• Recall that “unbundling” is needed for both EAD100LA fibre and for MPF copper

• 3 exchanges had to come out of plan and schools which were going to be linked have had to switch to more expensive EAD10/100 with also increase in excess dig costs

Obstacles and consequences - exchanges and core network

and indeed to capacity in general...more later.

• Firstly exchanges had to be “handed over” and then “backhaul” lines between exchanges installed / connected.

• Recall from July 18th meeting specific problems with this

Obstacles and consequences - exchanges and core network

30th Sept deadline and interim strategy

• Sharp focus since June once information about delays became clearer to formulating an “interim strategy” to ensure that all schools were transitioned by 30th Sept

Interim ADSL• In order to keep the rollout on schedule whilst these BT

Openreach issues were being resolved, it was decided to install an interim ADSL service into a number of schools.

• This service aims to provide a minimum of 2Mb download speed  (and up to 17Mb) and approx 600K upload speed. This is likely to be a higher capacity than the actual current usage levels.

• The service, which is being installed in a number of primary schools, ensures a continuity of service until the new main circuits are installed in the near future and reduces the risk of the workstream falling behind the planned completion date.

• By making these adjustments, the programme is still on track for all Surrey schools to have undergone transition by 30th September 2011.

Slide from 18th July presentation

• Orders have been placed for all schools, either for their “final” line and/or for the interim

What have we accomplished - connections into schools

Obstacles

Problems with BT Openreach arising from both their

systems / procedures and what seems to be a lack of

capacity in Surrey for new broadband lines at each stage of

the link. Some delays caused by upgrades by BT to their

“spine” which are not charged to the project but do cause

delays

Nature of BT Openreach delays –Secondary Schools

• Howard of Effingham School– When order placed, BT came back and reported in July that the ducting had

been physically damaged by a third party, and so the ducting itself would need repoairing, requiring road closure. Ducting ECD 9/9, Spine work ECD 16/9 and CDD 30/9, DSO escalation in place, overtime / weekend working is agreed and BT will carry out where possible to recover the lost time and bring the current completion dates forward. Now report that it will take weeks to feed new fibre through

– Contingency order placed for MPF 4-pairs, F&T scheduled for 27/9 • The Bishop Wand C of E Secondary School

– BT issued scope of works for cabling work, revised CDD 14/9– Contingency order placed for MPF 4-Pairs, in planning, no F&T scheduled yet

• Broadwater County School– Transition date 14/9

• Thamesmead School– Transition date 13/9

• Actions: – KCOM/BT Openreach escalation / updates– Comms with schools – keep them informed

Nature of BT Openreach delays – Copper Circuit Delivery

• As at 8/9, 27(13%) of copper orders placed are still in copper complex planning (CCP)

• Actions: – KCOM/BT Openreach review twice a week– Weekly detailed report of copper sites in complex

planning

Other BT Openreach problems and delays

• PSTN lines dead (over 40 schools)• MPF Low Bandwidth : a number of schools

are receiving around 4Mbps on a 4-pair MPF symmetric service instead of the estimated 10Mbps. 

– As at 9/9,

310 schools transitioned to new RM service• 41 Secondary, 260 Primary+Special+etc,

County Council, 4S, 7 – cancelled

– Issues• BT Openreach delays, schools, High PSTN

line failures

What have we accomplished - transitions

Top 5 Reasons for school Transition Failures

• Planned for 5% of school transitions to have failure/delays

• Current feedback shows 20%

• Top 5 failures/delays are (1 & 2 approx 80%):1. PSTN line dead or not found2. ADSL download bandwidth too low (< 2 Mbps)3. DOA/Wrong Router or Smartcache set-up delay4. Environmental issues5. Engineer error / delayed

2. (ADSL download bandwidth too low - situation much

improved now with “3 day delay” technique

Easynet Transition Status

• All cease requests raised on school broadband primary and backup circuits

• All Easynet services to be ceased by 30th September

• SMTP Feeds ready to use

• All Easynet additional services ceased– Officemail– Web hosting– SMTP feeds

Excess Dig Costs Status

• Budget £300K - £500K• Actual to 18th July = £236K• Budget review on 22/7 and during August• Significant increase caused by “no choice” changes e.g.

exchanges being taken out of plan, thus leading to more fibre connections

• Actual to date £425K

• Budget review by Surrey Broadband Implementation Project Board after Sept 30th and then discuss at next meeting of ICT Board

30th September

• Our absolute top priority is to ensure that all sites have transitioned by 30th September

• Actions to achieve this: proposed date for remaining schools NOT transitioned (moving to up to 12 installs/transitions per day)

• W/c 12/9 – Total 60: 47 confirmed, 13 proposed • W/c 19/9 – Total 21: 12 confirmed, 8 proposed, 1 cancel• W/c 26/9 – Total 13: 4 confirmed, 9 proposed

Ongoing contact with schools to get

confirmation of date from them

Implementation Schedule - Rules• Moving to up to 12 installs/transitions per day

– Preference for main broadband connection and backup

• All Interim ADSL dates before 16/9 and 2 weeks before cease date – Failures, fixed and revisit by end September

• All Secondary schools transitioned by w/e 16/9 excluding Howard

• Split sites transitioned by w/e 23/9 excluding Cranleigh sites– Main broadband connection at Master site, as priority– Interim ADSL for Master and/or Satellite sites using “Terminal Services”.

Approach proposed by Babcock– Pre-check visit for proposed Interim ADSL sites due to complete on 14/9

Implementation Phasing

(*) – Pilot Site Installs

Subject to BT delivery dates and schools access

  May June July Aug Sept Cancelled Total

Planned 6 129 156 38 72 6 404

  Transitions Up to 12 per day

  As at 9/9 Rollout Interim ADSL

Completed 3* 44 121 73 62 7 310

Forecast 0 0 0 0 93 1 94

End Month 3 44 121 73 155 8 404

Total 3 47 168 241 396  

Overdue 3 88 123 88 0    

Key Communications

– Bulletin• Timescales and BT Openreach delays• Interim ADSL• Status of project

– www.SurreyTechs.info

– Split sites use of “terminal services” for Interim ADSL connection at satellite

– Phase 2 / revisits to updated broadband connection

Post-transition problems

• Mainly around SmartCache• Meeting RM / 4S this week• SmartCache workshop in next few weeks - date to

be confirmed• Also aware that some schools have experienced

outages because they only have one line connected currently (e.g. with Interim ADSL, that will become backup once main line installed), and so if fault in their one line, problem as no backup

Questions

• At this stage, on what has happened• Second part of presentation is on “post 30th

Sept”, with questions at end

Post 30th Sept

• Need to take stock once transition complete to RM service, including actual bandwidth

• Need to do careful financial review (Mark Bisson is liaising with Rhona Barnfield, Chair of Schools Forum) - next meeting on Schools Forum on Oct 6th (date tbc)

• Capital / set-up costs• Excess dig costs• Revenue costs - funding to and charging of schools

Next meeting of ICT Board

• Suggest a date mid-Oct• A set of options will be prepared for the

meeting within the financial constraints set by Schools Forum

• Need to develop some criteria to underpin decisions

BAU Support

• 4S engineering

• Help desk contact for transition:– 0800 073 4444 (option 1 and 1 again)

• RM SEGfL Service Desk for BAU:– 0845 121 3348

Support and Documentation www.surreytechs.info

– Environment requirements– Broadband Transition Responsibilities document– Smartcache2 sites– Smartcache2 and installation, filtering configuration and

background documentation– School List with phasing schedule/timings and circuit types– Exchange details– Important addresses for schools

• Proxy, DNS Forwarders, SMTP feed– Site Acceptance Form– School transition works form (who will do work for school)

Summary

• Very important to continue to work with schools;  – Individual schools can talk with 4S reps re their own

situation

• Schools NOT yet transitioned are being contacted now to confirm date

• Important to remember what has been achieved in the face of major obstacles

• Still on track for all schools transitioned by 30th September