blog.factual.com/introducing-drake-a-kind-of-make-for-data, posted 2013 by peter in automation conversion development free
Paul Butler, a self-described Data Hacker, recently published an article called “Make for Data Scientists“, which explored the challenges of managing data processing work. Paul went on to explain why GNU Make could be a viable tool for easing this pain. He also pointed out some limitations with Make, for example the assumption that all data is local.
We were gladdened to read Paul’s article, because we’d been hard at work building an internal tool to help manage our data workflows. A defining goal was to end up with a kind of “Make for data”, but targeted squarely at the problems of managing data workflow.
The basic idea of story branching (sometimes referred to as “issue-driven development”) is that you create a development branch for each and every JIRA issue you implement.
Bug fixes, user stories, spikes… they all get their own branch.
Madness, you say!
And I would agree with you if we were still using a centralized version control system like SVN.
But branching in Git is very lightweight and merges don’t lock up the entire repository, making this crazy idea quite practical.
HTTP already has its own authentication system, and there are hundreds — perhaps thousands — of tools that know how to work with it. Wouldn’t it be great if there was a way to use that in our own scripts, but have something communicate with the services using OAuth behind the scenes? Thus, foauth.org was born.
Rather than try to build a bunch of bells and whistles and make everything really complicated, we focused on just one task: taking OAuth out of the equation when accessing your own data. So, unlike Apigee, we’re not monitoring your API usage or promising any statistics or anything like that. Our goal is to help you login with OAuth-compliant services using HTTP Basic authentication. That’s it.
Ansible is a radically simple model-driven configuration management, deployment, and command execution framework. Other tools in this space have been too complicated for too long, require too much bootstrapping, and have too much learning curve. Ansible is dead simple and painless to extend. For comparison, Puppet and Chef have about 60k lines of code. Ansible’s core is a little over 1000 lines.
Where Ansible excels though, is expressing complex multi-node deployment processes, executing ordered sequences on different sets of nodes through Playbooks. Playbooks contain one or more plays, each executed against a different batch of nodes. Think about webservers, database servers, and backend servers in a multi-node web environment. A play can address each set of machines in a cycle, ensuring the configurations of the machines were correct and also updating them to the specified version of software if required.
Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.
Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool.
Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). Sikuli includes Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API's support. You can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iphone or android application running in a simulator or via VNC.
Rundeck is an open-source process automation and command orchestration tool with a web console.
Dispatch shell commands and scripts across all your physical or virtual nodes from a web-based or command-line interface. Automate ad-hoc and routine procedures. Use the API and plugins to integrate with other services. Use your LDAP/AD for authentication, and configure extensive access control.
The WilliamsWarn Personal Brewery has six main technical features that, when combined together, create the world's first all-in-one brewing appliance. This combination then allows you to make the freshest beer on the planet with minimum effort.
This thing looks ridiculously cool. Makes me wonder though... I'm sure a total noob (like me) can use it to make okay beer -- but would a skilled brewer (like I would want to become if I had one of these) be able to make excellent beer with it? Then I'd want one. Or at least I would if I had somewhere to put it. It's pretty stylish, but not something I'd want in my living room.