Virtual Hosting and Automation Take It to the Bank for Barclays
The Old Days
Barclays was satisfied with their website operations: waterfall-style software projects, environments carefully provisioned manually – one at a time – and discrete physical hardware hosting each disparate piece of the operational stack. Capital expenses and personnel numbers grew exponentially. Provisioning even the simplest service required days to implement, but existing groups stayed with the program, preferring excessive provisioning and hiring more server-herders to the perils of automation and virtualization.
When Barclaycard became interested in a daily-deals style website requiring lean budgets and fast turnaround times, the old ways had to change.
Acquired by Barclaycard in June 2012, Analog Analytics was handed the project to build a daily-deals website yesterday. To automate high-traffic international e-commerce environments quickly, Analog Analytics selected Lee Whalen at Fuzzy Logic for his performance at Endurance International (very large national web-hosting organization) and Mimecast (distributed international email compliance company), as well as his proven ability to “automate all the things”.
Coming on board in December 2013, Fuzzy Logic set to work automating software project deployments using Ansible, and later, provisioning infrastructure through a custom-built Fog library command-line application linking directly to the Openstack API. This allowed the team to spin up fully-provisioned environments – over a dozen hosts, support networks, and connections to load balancers – in about two hours.
Negotiating resistance was as much part of the job as the systems work. At the time, Barclays preferred the known comforts and limitations of the staid waterfall methodology to flexibility, self-healing, idempotence, and scaling; they had to be wooed in the universal language of money.
Virtual Hosting Speaks
In the beginning, static production, staging and development environments for the team were housed in multiple (expensive) racks of leased hardware, with each piece of the stack on its own physical host. No virtual hosts allowed.
To help Barclays break this costly hosting habit, the team worked first to shift development to a virtual environment. Costs were a huge concern: with both hosting (CapEx) and the team’s hourly rates (OpEx) spiraling, the Compliance team was receptive to using a virtual environment in development, where there would be no live user traffic.
The proof of concept was built with Rackspace and their ‘Private Cloud’ offering. Once the team demonstrated the Vagrant Openstack connector spinning VMs up and down into pre-defined virtual networks, exhibiting increased productivity and flexibility, Compliance green-lit building out additional Openstack clusters.
Ultimately, the project saved $1.4 million per year in hosting costs alone. When the site was up for accounting review in 2014, that $120,000 a month savings allowed the team to keep the project alive, delivering well beyond the expectations of management.
Fuzzy Logic and the Analog Analytics team were eager to implement DevOps concepts like rapid deployment, short iterations, and automated testing and deployment, but just as they were ready to begin rapid sprints to build, test, and implement a flexible and highly responsive promotions website, they hit a roadblock.
Barclays security, physically located in the UK and wholly anonymous, insisted on maintaining the old, familiar, management style, including two-week turnaround times for change review.
Every activity was suspect. The team heard decrees such as:
Staging virtual machines must be provably shown to not share hardware with production machines.
Developers are forbidden to manipulate production servers.
Infrastructure virtual machine changes, including creating new VMs and installing utility packages, must be reviewed via lengthy change request procedures.
Navigating these deep bureaucratic waters required the combined efforts of Fuzzy Logic principal Lee Whalen – who had become fluent in myriad, sometimes conflicting, compliance requirements from his experience at Mimecast – and the AA team manager. A system was devised that allowed all required change requests to be submitted, in advance, to comply with the two-week deadline. Consistent, continued application of the counter-flood won a dispensation to file ex post facto CR documentation, effectively speeding deploys to multiple times per day.
Other wins that helped the team meet their goals included:
- Coaxing access to the production logs, streamed off-VM to a centralized server, in order to pull and sanitize a production database copy to feed into staging and development.
- Negotiating creation of a sandbox where the team could build and destroy networks, VMs, and servers at will, which kept the security team calm and relaxed while the project advanced.
After significant initial push-back, Fuzzy Logic and Analog Analytics won over Barclays Security with a combination of compliance savvy, persistence, and a consistently stable environment, as well as reflected glory from fast feature releases. Ultimately, they earned the security team’s trust, enabling them offer new features in a matter of days rather than the rumored six-month lead times for internal teams.
Fuzzy Logic was able to:
- Automate the physical and virtual host management within a strict compliance environment for the Barclaycard Bespoke Offers portal based on CentOS, Rails 3, Unicorn, Nginx, Chef, and Redis/Resque.
- Save over $120k/ month in hosting fees by providing key support for an environment-wide migration from Rackspace fully managed hardware to a privately hosted Rackspace OpenStack.
In addition, Fuzzy Logic supported the team by:
- Writing extensive and detailed documentation to reduce new DevOps teammates’ onboarding from two weeks to two days.
- Co-developing a Vagrant-based provisioning to spin up complex environ- ments with a single command, reducing a multi-day, highly manual process to two hours.
- Co-developing a FogLibrary-based provisioning tool so that the same code and JSON-based configuration files could deploy independent of the cloud provider backend.
The Bespoke Offers portal came online before the end of 2013, after only a month in development. It traded from 2013 until October 2015, when Barclaycard announced its closure.
Based in Seattle, Washington, Fuzzy Logic serves clients worldwide, assisting development and operations teams in leveraging the power of the latest technologies in virtualization, server administration, communication, and deployment to deliver solid products that are efficient, faster to market, and customer-friendly.
Fuzzy Logic’s clients operate in diverse fields such as digital media, data analytics, healthcare, social networking, e-commerce, and banking.
Analog Analytics was a provider of a white-label software platform for digital coupons and daily deals promotions. Sold to Barclaycard in May 2012, the company’s technology