DAVID A. STAINTON's
Operations Engineer/Security Advisor
Berlin (X - April 22nd 2014)
- configured new chef system to manage core infrastructure
- conducted datacenter migration
- conducted Postgres DB migration/upgrade
- documented contingency plans
- increased high availability of various services
- assisted teammates with writing chef cookbooks and other operational tasks
San Francisco (August 2013 - November)
San Francisco (February - May 2013)
Worked on Chef 11 migration, Redis upgrade, postfix migration, Chef Berkshelf release/deploy process etc.
San Francisco (May 2012 - September 2012)
- Dealt with technical debt.
- Performed many database migrations.
- Assisted in Chef migration.
San Francisco (October 2010 - March 2012)
San Francisco (July 2010 - October 2010)
- Wrote Puppet manifests for Hadoop and Hive.
- I was responsible for Hadoop cluster upgrades.
- Created the Mysql 5.1 xtradb innobackupex snapshot backup system for Digg's Mysql databases.
- Worked with Clusto, Scribe and Mogile FS
- Wrote tools to assist engineers with using Cassandra; I wrote Cassandra Page-cache Usage in Python.
San Francisco (2008 - August 2009)
- 40+ servers with hardware managed-hosting (i.e. not colocation and not managed-hosting)
- Rolling upgrades to the Mysql clusters
- Live migration from ServerBeach to Softlayer managed hosting
- Operations and capacity planning of a large Mysql clusters...
- Database operations automated via Python
- Continually improving monitoring systems (Nagios and Ganglia)
- On-call rotation
- Cluster configuration management with Bcfg2 (similar to Puppet)
- Automatic Debian package creation
Linux Systems Administrator
San Francisco (April 2006 - August 2007)
- Network Administration: fixed network outages, setup loadbalancing traffic to webservers via HAProxy and STunnel,
customized IP security policies and routing schemes with a Juniper Netscreen firewall, Linux IP Tables and Cisco routers
Systems: "shadow DNS" via Daniel J. Bernstein's TinyDNS, mail via Postfix, SpamAssassin and qpsmtpd with a custom Perl plugin,
Apache, Tomcat, MySQL, Squid, Mon (service monitoring system) w/ custom Perl scripts, backup system via cron scripts
- Applications: VMWare Server setup with Windows and Linux images, Cruise Control - Java continuous build server
Site Operations Engineer
San Francisco (November 2004 - August 2005)
Worked on call, responsible for site reliability and uptime.
Facilitated in site downtime maintenance including a timed
bounces of various services for site operation, collaborating with the
Wrote various monitoring and database analysis tools in Java, Perl and
Bash. This includes usage of Gnuplot and Perl to generate web viewable
graphs, Mon monitor's which send alerts to the pager, a Java program to
automatically fix failed website orders...
Ran many SQL queries on various Oracle database nodes to maintain
coherency for Snapfish accounts and photo albums.
Performed various systems administration tasks such as maintain crontabs,
configure postfix, custom compiled apache etc.
Responsible for the management and generation of all internal SSL
certs. Used perl to automate the process.
Reported upon site bugs and collaborated with the site developers to get
contract Tech Support Engineer / Developer,
Cupertino (May 2004 - July)
- Serviced tech support calls troubleshooting firewalls, routing problems, DNS and
mail server configurations.
- Automated tasks in perl and setup SNMP monitoring.
contract Software Quality Assurance Engineer,
DataDomain, Palo Alto
(January 2004 - April)
Performed hardware/software manual tests.
Wrote Perl programs to automate testing of DataDomain's software.
Installed and configured Linux systems to run QA tests of DataDomain's backup server.
Debugged program crashes/core dump analysis with GDB (the GNU Debugger).
Created performance tests to ensure the Data Domain Restorer met product SLA.
Network Tools Developer, Northpoint Communications, San Francisco
(April 1999 - December 2000)
- Deployed custom monitoring software to Solaris/Linux servers in less than 50 points of presense...
- Wrote tools to automate the
monitoring and maintenance of the network.
Some of these tools were used to modify
the configuration of hundreds of Cisco routers, Ascend switches,
automate configuration of MRTG (multi router traffic grapher) etc.
Worked in a project writing in-house software
to automate the testing of copper loops (for DSL) from circuit orders.
I programmed the backend code and
collaborated with database and web CGI programmers.
Configured CopperMountain DSLAMs, Cisco routers and
Ascend frame-relay switches
Network Engineer / Tools Developer, CRL Network Services,
San Francisco (December 1997 - October 1998)
- Worked with hundreds of cisco routers
Wrote various monitoring facilities for the Network Operations Center,
including a program which displayed packet loss to peer networks from
Worked extensively with MRTG (multi router traffic grapher)
Performed frame-relay trouble shooting on Ascend switches (via HPOpenView)
and EIGRP/BGP trouble shooting on Cisco routers.
Set up Cisco Flow Analyzer, MIT Kerberos
V4 on a SunOS server, including serious Sendmail configuration,
modified the C source code for the standard local mailer tranfer agent to
include Kerberos authentication and modified the C source code of the
Merit AAA RADIUS
server for special username parsing.