words and images by mk swanson
 
Virtual Hosting and Automation Take It to the Bank for Barclays – Case Study

Virtual Hosting and Automation Take It to the Bank for Barclays – Case Study

The Old Days

Barclays was satisfied with their website operations––waterfall-style software projects, environments provisioned one at a time, and discrete physical hardware hosting each disparate piece of the operational stack. Capitol expenses and personnel numbers grew exponentially, and provisioning took three days, but existing groups stayed with the program, preferring excessive provisioning and hiring more server herders to the perils of automation and virtualization.
When Barclaycard became interested in a daily deals style website requiring lean budgets and fast turnaround times, the old ways had to change.

Website Now

Acquired by Barclaycard in June 2012, Analog Analytics was handed the project to build a daily deals website yesterday. To automate high-traffic international environments quickly, Analog Analytics selected Lee Whalen at Fuzzy Logic for his performance at Endurance International and Mimecast, as well as hi sproven ability to “automate all the things.”
Coming on board in December 2013, Fuzzy Logic set to work automating software project deployments using Ansible, and later, provisioning infrastructure through a custom-built Fog library command-line application linking directly to the Openstack API. This allowed the team to spin up fully-provisioned environments, including a dozen hosts, support networks, and connections to load balancers, in about two hours.
Negotiating resistance was as much part of the job as the systems work. Barclays preferred the comfort of the known to the advantages of flexibility, efficient error-handling, idempotence, and scaling; they had to be wooed in the universal language of money.

Virtual Hosting Speaks

In the beginning, production, staging and static development for the team were housed in four racks of hardware, with each piece of the stack on its own physical host. No virtual hosts allowed.
To help Barclays break this expensive hosting habit, the team worked first to shift development to a virtual environment. Costs were a huge concern, with both hosting (CapEx) and the team’s hourly rates (OpEx) spiraling, so the compliance team was receptive to using a virtual environment in development, where there would be no live user traffic.
The proof of concept was built with Rackspace. Once the team demonstrated the Vagrant Openstack connector spinning VMs up and down into pre-defined virtual network environments and documented increased productivity and flexibility, compliance green-lighted building out additional Openstack clusters. 
Ultimately, the team saved $1.4 million per year in hosting costs alone. When the project was threatened in 2014, that $120,000 a month savings allowed the team to keep the project alive, delivering well beyond the expectations of management.

Compliance Complications

Fuzzy Logic and the Analog Analytics team were eager to implement DevOps concepts like rapid deployment, short iterations, and automated testing and deployment, but just as they were ready to begin rapid sprints to build, test, and implement a flexible and highly responsive promotions website, they hit a roadblock.
Barclays security, physically located in the UK and wholly anonymous, insisted on maintaining their old project management style, including two-week turnaround times for change review.
Every activity was suspect. The team heard decrees such as:

Staging virtual machines must be provably shown to not share hardware with production machines.
Developers are forbidden to manipulate production servers.
Infrastructure virtual machine changes, including creating new VMs and installing utility packages, must be ordered via change request.
Navigating these deep bureaucratic waters required the combined efforts of Fuzzy Logic principal Lee Whalen, who had become fluent in myriad, sometimes conflicting, compliance requirements from his experience at Mimecast, and the AA team manager. They completed allrequired product deploy change requests (more than a dozen every day) in time for the two-week turnaround. Continued application of the counter-flood won a dispensation to file ex post facto CR tickets, allowing the team to deploy twice a day.
Other wins that helped the team meet their goals included:

  • Coaxing access to the production logs, streamed off-VM to a centralized server, in order to pull and sanitize a production database copy to feed into staging and development.
  • Negotiating creation of a sandbox where the team could build and destroy networks, VMs, and servers at will, which kept the security team calm and relaxed while the project advanced.
    After significant initial push-back, Fuzzy Logic and Analog Analytics won over Barclays security with a combination of compliance savvy, persistence, and a consistently stable environment, as well as reflected glory from their fast feature releases. Ultimately, they earned the security team’s trust, enabling them offer new features in a matter of days rather than the rumored six-month lead times for internal teams.

Big Wins

Fuzzy Logic was able to:

  • Automate the physical and virtual host management within a strict compliance environment for the Barclaycard Bespoke Offers portal based on CentOS, Rails 3, Unicorn, Nginx, Chef, and Redis/Resque.
  • Save over $120k/ month in hosting fees by providing key support for an environment-wide migration from Rackspace fully managed hardware to a privately hosted Rackspace OpenStack.
    In addition, Fuzzy Logic supported the team by:
  • Writing extensive and detailed documentation to reduce new DevOps teammates’ onboarding from two weeks to two days.
  • Co-developing a Vagrant-based provisioning to spin up complex environments with a single command, reducing a multi-day, highly manual process to two hours.
  • Co-developing a FogLibrary-based provisioning tool so that the same code and JSON-based configuration files could deploy independent of the cloud provider backend.
    The Bespoke Offers portal came online before the end of 2013, after only a month in development. It traded from 2013 until October 2015, when Barclaycard announced its closure.

Fuzzy Logic

Based in Seattle, Washington, Fuzzy Logic serves clients worldwide, assisting development and operations teams in leveraging the power of the latest technologies in virtualization, server administration, communication, and deployment to deliver solid products that are efficient, faster to market, and customer-friendly.
Fuzzy Logic’s clients operate in diverse fields such as digital media, data analytics, healthcare, social networking, and banking.
[www.fuzzylogic.com]

Analog Analytics

Analog Analytics was a provider of a white-label software platform for digital coupons and daily deals promotions. Sold to Barclaycard in May 2012, the company’s technology powered the Barclaycard’s Digital Marketplace “Bespoke Offers” program from 2013 through 2015.

Barclaycard

Barclaycard is a global payment business, part of Barclays Retail and Business Banking, located in Northampton, UK. It is the leading UK credit card issuer, with over 10 million UK customers.
[https://www.home.barclaycard/]

Links for images

Bespoke Offers: [https://www.bespokeoffers.co.uk/]
Analog Analytics: [http://www.versatile-connections.com/analog-analytics—case-study]