Videology | DevOps

Videology (www.videologygroup.com) is thriving in an industry that changes as fast as consumer tastes. Founded in 2007, the privately held company is now one of the world’s leading video advertising platforms, helping advertisers effectively reach consumers around the globe wherever they watch video. As of early 2016, Videology employed over 300 people in 28 countries. The company also relies upon its partner Coherent Solutions to supplement internal staff with knowledgeable professionals who can scale up and down to meet project needs.

With hundreds of customers and thousands of video advertising options, Videology’s complex applications are high-load, enabling customers to plan, buy, optimize and measure digital advertising campaigns. There are also complex analytics for scrutinizing audience characteristics, revenue streams and more.  Needless to say, the potential for error is high as all this technology is being deployed across a labyrinth of servers by dozens of users at any one time.

So in early 2014, Videology began experimenting with DevOps in an effort to create more stable, scalable apps. Recognizing its potential, they brought on Joseph Julian (Jules) as the Director of Engineering to establish the company’s DevOps program later that year.

“The company wanted to reduce downtime and optimize its infrastructure,” explained Jules. “Efficiency, speed, accuracy – those were the primary goals. Saving money was expected as a by-product, but it wasn’t the primary goal.”

Server usage cut in half, saving tens of thousands of dollars a month

Yet save they did – to the tune of tens of thousands of dollars monthly. But it took time. Jules had to build his team from resources that came from both systems administration and software development. They were eager to learn, but required months of extensive training. Eventually Jules was able to seed each of them into Videology’s various development groups.

Meanwhile, Jules was also reviewing contingent staff which included four resources with an Asian supplier and one DevOps resource at Coherent Solutions’ Minsk, Belarus development center.

“The guy in Minsk was putting out ten times as much work as any one of the other guys,” Jules discovered. “So I let that firm go and built my offshore team at Coherent.” Videology’s DevOps group at Coherent is up to six people as of this writing. They have all been seeded into the development groups as well.

“I don’t draw distinctions between our onshore and offshore resources,” explained Jules. “With about a half a day of overlap, our Minsk people can participate in ceremonies via Skype, Zoom and HipChat.” Jules also noted that the Minsk team works at odd hours and on the weekends to accommodate the needs of Videology’s 24/7 systems.

With the department stabilizing, Jules was able to concentrate more time on reviewing Videology’s complex infrastructure which included hundreds of instances on the AWS cloud, processing over eleven billion requests a day. While all these servers were needed during peak usage, many of them were idle for large parts of the day.

Jules saw an opportunity to optimize usage through auto-scaling. “By redesigning our server configuration we don’t pay for what we aren’t using. Our configured deployments can do in one to two hours what two to three people could process in a day before. And it has reduced failures by at least fifty percent.”

Just as impressively, instance usage is down significantly, saving Videology tens of thousands of dollars per month. And each of the DevOps support teams has reached a full week of zero failures – earning a steak dinner for the team. A few teams have made it a full month without anyone’s pager going off.

The benefits of DevOps at Videology just keep coming. The development teams are more engaged with operations as they see its advantages. Many developers now attend DevOps’ system design reviews to learn more.  In fact, more than half of Jules’ DevOps group were originally full-time developers.

“Some of our most interesting work is being done in DevOps,” Jules said proudly. “We are tracking trends by platform and environment, and driving towards a fully automated ecosystem. We’re rolling out automated integration testing with an end goal of continuous delivery.  It could happen this year for one or more of our teams.”