Thursday, August 22, 2013

Angular AJAX Upload

Though the Internet would have you believe otherwise, uploading a file asynchronously from AngularJS isn't that hard. I don't want fancy colors or previews or progress bars or any of that. I want to upload a file from my AngularJS-backed webapp without reloading the page. Also, I don't care about old browsers. If you do, then this might not work for you.

After struggling with blueimp's library for way too long, I decided to just implement the part I needed.

Uploading a file using AJAX + AngularJS requires three things:

  1. AJAX
  2. AngularJS
  3. AJAX + AngularJS

1. AJAX

function upload(url, file) {
var formdata = new FormData(),
xhr = new XMLHttpRequest();

formdata.append('myfile', file);

xhr.onreadystatechange = function(r) {
if (4 === this.readyState) {
if (xhr.status == 200) {
// success
} else {
// failure
}
}
}
xhr.open("POST", url, true);
xhr.send(formdata);
}

The file will be posted to the server as the parameter named myfile.

2. AngularJS

app.directive('fileChange', function() {
return {
restrict: 'A',
link: function(scope, element, attrs) {
element.bind('change', function() {
scope.$apply(function() {
scope[attrs['fileChange']](element[0].files);
})
})
},
}
})

If you use the above directive like this:

<input type="file" file-change="runSomething">

when the user chooses a file to upload, runSomething will be called with a FileList. You can pass the first element in that list as the second arg to the upload function above.

3. AJAX + AngularJS

I can't provide a complete demo (because this blog isn't backed by a server I control). But this will probably get you really close:

<!DOCTYPE html>
<html lang="en">
<body ng-app="myapp" ng-controller="UploadCtrl">
<input type="file" file-change="upload">

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.min.js"></script>
<script>
// the javascript
var app = angular.module('myapp', []);

//
// Reusable Uploader service.
//
app.factory('Uploader', function($q, $rootScope) {
this.upload = function(url, file) {
var deferred = $q.defer(),
formdata = new FormData(),
xhr = new XMLHttpRequest();

formdata.append('file', file);

xhr.onreadystatechange = function(r) {
if (4 === this.readyState) {
if (xhr.status == 200) {
$rootScope.$apply(function() {
deferred.resolve(xhr);
});
} else {
$rootScope.$apply(function() {
deferred.reject(xhr);
});
}
}
}
xhr.open("POST", url, true);
xhr.send(formdata);
return deferred.promise;
};
return this;
})


//
// fileChange directive because ng-change doesn't work for file inputs.
//
app.directive('fileChange', function() {
return {
restrict: 'A',
link: function(scope, element, attrs) {
element.bind('change', function() {
scope.$apply(function() {
scope[attrs['fileChange']](element[0].files);
})
})
},
}
})

//
// Example controller
//
app.controller('UploadCtrl', function($scope, $http, Uploader) {
$scope.upload = function(files) {
var r = Uploader.upload('/uploads', files[0]);
r.then(
function(r) {
// success
},
function(r) {
// failure
});
}
});
</script>
</body>
</html>

More

You can do more things like handle multiple files, monitor progress, preview images, etc... But if you don't need all that, and you are using modern browsers, this should do just fine.

Thursday, August 15, 2013

Practical event-driven programming with Python and Twisted

Introduction

A article from 2008 entitled Practical threaded programming with Python was posted to HN today. And I thought, "how would those examples look with Twisted?"

For a great explanation about how Twisted does concurrency, see krondo's Twisted Introduction. On to the code:

Hello World

The first example in the article demonstrates that threads have IDs. Since we're not using threads, the most equiavelent way to do the same thing with Twisted is to not use Twisted at all:

import datetime


def run(what):
now = datetime.datetime.now()
print '%s says Hello World at time: %s' % (what, now)


for i in range(2):
run(i)

Output:

0 says Hello World at time: 2013-08-15 13:45:17.164933
1 says Hello World at time: 2013-08-15 13:45:17.165442

Using queues

The next example shows first a serial approach and then a threaded approach to "grab a URL of a website, and print out the first 1024 bytes of the page." Here are the synchronous/serial and threaded versions.

I should note that I've modified them to get all the page (instead of the first 1024 bytes) and to print a hash of the content (so as not to clutter up this post). It's interesting that only apple.com and ibm.com return the same hash every time.

Synchronous version

import urllib2
import time
import hashlib

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

start = time.time()
#grabs urls of hosts and prints first 1024 bytes of page
for host in hosts:
url = urllib2.urlopen(host)
print hashlib.sha1(url.read()).hexdigest(), host

print "Elapsed Time: %s" % (time.time() - start)

Output:

2430771cc3723e965b64eda2d69dd22b697dd4a0 http://yahoo.com
790ace256c1b683a585226d286859f9f2910d9b0 http://google.com
63fbbe761817ebef066f9562e96209ca25a6f0b3 http://amazon.com
dd2f34c7c4f47b49272d7922e4f17f7c1cafd3aa http://ibm.com
562ffc06504dc0557386524b382372448d6e953a http://apple.com
Elapsed Time: 3.34798121452

Threaded version

#!/usr/bin/env python
import Queue
import threading
import urllib2
import time
import hashlib

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

queue = Queue.Queue()

class ThreadUrl(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue

def run(self):
while True:
#grabs host from queue
host = self.queue.get()

#grabs urls of hosts and prints first 1024 bytes of page
url = urllib2.urlopen(host)
print hashlib.sha1(url.read()).hexdigest(), host

#signals to queue job is done
self.queue.task_done()


start = time.time()
def main():
#spawn a pool of threads, and pass them queue instance
for i in range(5):
t = ThreadUrl(queue)
t.setDaemon(True)
t.start()

#populate queue with data
for host in hosts:
queue.put(host)

#wait on the queue until everything has been processed
queue.join()

main()
print "Elapsed Time: %s" % (time.time() - start)

Output:

562ffc06504dc0557386524b382372448d6e953a http://apple.com
fb6fe32cb270f7929157bec5f29ee44f729949fd http://google.com
dd2f34c7c4f47b49272d7922e4f17f7c1cafd3aa http://ibm.com
3643a39f4dd641a3c08f8e5c409d0f5bc6407aed http://amazon.com
3072477b1680fc2650d9cb0674e5ef7972873bf6 http://yahoo.com
Elapsed Time: 1.23798894882

Twisted version

Here's one way to do the same thing with Twisted:

from twisted.internet import defer, task
from twisted.web.client import getPage
import time
import hashlib

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

start = time.time()

def printHash(content, host):
print hashlib.sha1(content).hexdigest(), host


def main(reactor, hosts):
dlist = []
for host in hosts:
d = getPage(host)
# when we have the content, call printHash with it
d.addCallback(printHash, host)
dlist.append(d)

# finish the process when the "queue" is done
return defer.gatherResults(dlist).addCallback(printElapsedTime)


def printElapsedTime(ignore):
print "Elapsed Time: %s" % (time.time() - start)


task.react(main, [hosts])

Output:

188eecd4da73515a9d1b3fde88d81ccc3a1e6028 http://google.com
562ffc06504dc0557386524b382372448d6e953a http://apple.com
dd2f34c7c4f47b49272d7922e4f17f7c1cafd3aa http://ibm.com
968fc83c1c7717575af03d43b236baf508134d0f http://yahoo.com
90c51ab729261bb72db922fb5ad22c0ae33c09da http://amazon.com
Elapsed Time: 1.36157393456

The run times of the threaded version and the Twisted version are comparable. Running them each multiple times, sometimes the threaded version is faster and sometimes the Twisted version is faster. They are both consistently faster than the synchronous version. Either way, this isn't a great benchmark and doesn't say much about how ansynchronous v. threaded will work in your particular case.

Working with multiple queues

The article's third bit of code shows how to use multiple queues to get the URL's body in one thread, then process it in another thread.

Threaded version

import Queue
import threading
import urllib2
import time
from BeautifulSoup import BeautifulSoup

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

queue = Queue.Queue()
out_queue = Queue.Queue()

class ThreadUrl(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, queue, out_queue):
threading.Thread.__init__(self)
self.queue = queue
self.out_queue = out_queue

def run(self):
while True:
#grabs host from queue
host = self.queue.get()

#grabs urls of hosts and then grabs chunk of webpage
url = urllib2.urlopen(host)
chunk = url.read()

#place chunk into out queue
self.out_queue.put(chunk)

#signals to queue job is done
self.queue.task_done()

class DatamineThread(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, out_queue):
threading.Thread.__init__(self)
self.out_queue = out_queue

def run(self):
while True:
#grabs host from queue
chunk = self.out_queue.get()

#parse the chunk
soup = BeautifulSoup(chunk)
print soup.findAll(['title'])

#signals to queue job is done
self.out_queue.task_done()

start = time.time()
def main():

#spawn a pool of threads, and pass them queue instance
for i in range(5):
t = ThreadUrl(queue, out_queue)
t.setDaemon(True)
t.start()

#populate queue with data
for host in hosts:
queue.put(host)

for i in range(5):
dt = DatamineThread(out_queue)
dt.setDaemon(True)
dt.start()


#wait on the queue until everything has been processed
queue.join()
out_queue.join()

main()
print "Elapsed Time: %s" % (time.time() - start)

Output:

[<title>Apple</title>]
[<title>Google</title>]
[<title>IBM - United States</title>]
[<title>Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs &amp; more</title>]
[<title>Yahoo!</title>]
Elapsed Time: 1.65801095963

Twisted version

For this simple example, it makes sense to just do the processing right after receiving the body. That would look like this:

from twisted.internet import defer, task
from twisted.web.client import getPage
import time
from BeautifulSoup import BeautifulSoup

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

start = time.time()

def printTitle(content, host):
soup = BeautifulSoup(content)
print soup.findAll(['title'])


def main(reactor, hosts):
dlist = []
for host in hosts:
d = getPage(host)
# when we have the content, call printTitle with it
d.addCallback(printTitle, host)
dlist.append(d)

# finish the process when the "queue" is done
return defer.gatherResults(dlist).addCallback(printElapsedTime)


def printElapsedTime(ignore):
print "Elapsed Time: %s" % (time.time() - start)


task.react(main, [hosts])

Output:

[<title>Google</title>]
[<title>Apple</title>]
[<title>IBM - United States</title>]
[<title>Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs &amp; more</title>]
[<title>Yahoo!</title>]
Elapsed Time: 1.80365180969

(As with the previous examples, neither the threaded nor the Twisted version are much different in speed.)

Hey!

"Hey! Those aren't the same!" I hear you say. You are right. They are not. The threaded version could extract the title in ThreadUrl.run instead of putting the content in queue for a DatamineThread.

I think the author was trying to show how you can make two threads work together on something... big? I haven't come up with a problem where it makes sense to write something in the Twisted version other than d.addCallback(printTitle, ...). If you have an idea post a comment, and I'll happily update this post (or make another post).

Conclusion

You can do things with threading. You can do things with Twisted. You should investigate Twisted (mostly for reasons not mentioned in this post). As noted above, krondo's Twisted Introduction is good, or there's some stuff I've written.

Also, if anyone can think of a better scenario for the two-kinds-of-thread-workers model, I'll update (or post again) with what a Twisted version might look like.

Wednesday, July 10, 2013

Angular injection

tl;dr is marked throughout by ∴

I don't like magical code. AngularJS is magical. I must fix this.

Dependency injection was one of AngularJS's first evil magicks I encountered. The idea that calling this function

function myFunction($scope, $http) {
...
}
will magically reach out to the universe and grab the correct values for $scope and $http runs contrary to all the JavaScript I've ever used. You can't do that!

So I dug in to discover the magicks. And now it's not magic! It's great! It's rougly equivalent to import in Python or require in Ruby. Here's how it works:

Modules

AngularJS groups injectable things together into modules. The following code will:

  1. make a module named woods
  2. add a provider to the woods module named Eeyore, which has a constant value
var woods = angular.module('woods', []);
woods.value('Eeyore', 'sad')

Here's some of the source for the module function plus context (see the full source here — the comments are helpful):

// from setupModuleLoader()
function ensure(obj, name, factory) {
return obj[name] || (obj[name] = factory());
}

// ...

var modules = {};
return function module(name, requires, configFn) {
// ...
return ensure(modules, name, function() {
// ...
var moduleInstance = {
// ...
requires: requires,
name: name,
provider: invokeLater('$provide', 'provider'),
factory: invokeLater('$provide', 'factory'),
service: invokeLater('$provide', 'service'),
value: invokeLater('$provide', 'value'),
constant: invokeLater('$provide', 'constant', 'unshift'),
filter: invokeLater('$filterProvider', 'register'),
controller: invokeLater('$controllerProvider', 'register'),
directive: invokeLater('$compileProvider', 'directive'),
// ...
};
// ...
return moduleInstance;
// ...
});
};

  1. The ensure(obj, name, factory) function makes sure that obj has an attribute named name, creating it by calling factory if it doesn't.
  2. The module(name, requires, configFn) function adds a moduleInstance named name to the global-ish modules object (by using ensure).

angular.module(...) adds a module to some global-ish module registry.

Injectors

Injectors find providers from among the modules it knows about. By default, AngularJS creates an injector through the bootstrapping process. We can also make an injector with angular.injector() and use it to access providers within modules:

// Run this in a JavaScript console (on a page that has AngularJS)

// Make a woods module with an Eeyore provider
var woods = angular.module('woods', []);
woods.value('Eeyore', 'sad')

// Make an injector that knows about the 'woods' module.
var injector = angular.injector(['woods'])

// Get poor Eeyore out of the module
injector.get('Eeyore');
// -> "sad"

The creation of injectors and how they know where things are is somewhat recursive (and the code is a little hard to read). I will unravel that magic in another post as it was making this post too long. For now, just know that

Injectors can find the providers you add to modules (e.g. through .value(...) or .factory(...)) and can find modules that were previously added to the global-ish module registry.

Invoke

Using an injector, we can invoke functions with dependency injection:

// Run this in a JavaScript console (on a page that has AngularJS)

// Make a woods module with an Eeyore provider
var woods = angular.module('woods', []);
woods.value('Eeyore', 'sad')

// Make an injector that knows about the 'woods' module.
var injector = angular.injector(['woods'])

// Imbue a function with sadness
function eatEmotion(Eeyore) {
return 'I am ' + Eeyore;
}
injector.invoke(eatEmotion);
// -> "I am sad"

But how does it KNOOooooowwwWWW??

How does AngularJS know the names of the arguments a function is expecting? How does it know that my weather function's arguments is named sunny?

function weather(sunny) {
...
}

That's an internal detail of weather, inaccessible from the outside, no? I've done introspection with Python, but this is JavaScript.

How AngularJS gets the argument names made me laugh out loud when I found it. It's a dirty (effective) trick found in the annontate function (full source):

var FN_ARGS = /^function\s*[^\(]*\(\s*([^\)]*)\)/m;
var FN_ARG_SPLIT = /,/;
var FN_ARG = /^\s*(_?)(\S+?)\1\s*$/;
var STRIP_COMMENTS = /((\/\/.*$)|(\/\*[\s\S]*?\*\/))/mg;
function annotate(fn) {
var $inject,
fnText,
argDecl,
last;

if (typeof fn == 'function') {
if (!($inject = fn.$inject)) {
$inject = [];
fnText = fn.toString().replace(STRIP_COMMENTS, '');
argDecl = fnText.match(FN_ARGS);
forEach(argDecl[1].split(FN_ARG_SPLIT), function(arg){
arg.replace(FN_ARG, function(all, underscore, name){
$inject.push(name);
});
});
fn.$inject = $inject;
}
} else if (isArray(fn)) {
last = fn.length - 1;
assertArgFn(fn[last], 'fn');
$inject = fn.slice(0, last);
} else {
assertArgFn(fn, 'fn', true);
}
return $inject;
}

If you pass a function to annotate it will convert that function to a string and use regular expressions to get the names of the arguments.

I should note, however, that the practice of depending on argument names for injection is discouraged (because of how the names get munged during minification). It makes the code look cleaner, though. Maybe we should work on changing minification to handle this introspective kind of injection.

Which functions have it? Which don't?

When you're just starting with AngularJS, it's a little frustrating that some functions are magic (i.e. are called with injection) and some are seemingly inert. For instance, when writing a directive, link is not called with dependency injection, but controller is.

The provider methods are called with injection (factory, value, etc...). And directive controllers are called with injection. From the official docs:

DI is pervasive throughout Angular. It is typically used in controllers and factory methods.

Sadly, the only way to know if a function is called with dependency injection is to... know. Read the docs or the source, and build up an ample supply of doing it wrong :)

Namespacing

Modules provided to an injector will stomp on each other's providers:

// Run this in a JavaScript console (on a page that has AngularJS)

function mineFor(Thing) {
return "I found " + Thing + "!";
}


// Make two modules that each define a Thing provider
var good_module = angular.module('good', []);
good_module.value('Thing', 'gold');

var bad_module = angular.module('bad', []);
bad_module.value('Thing', 'sour milk');

// Make an injector
var injector = angular.injector(['good', 'bad']);

injector.invoke(mineFor);
// -> "I found sour milk!"

I don't know if this is by design or if there are plans to address it. Be aware of it.

In summary

Dependency injection in AngularJS is roughly equivalent to other languages' including and importing, but scoped to functions. Some of the magic is accomplished by exploiting function.toString() and regular expressions.

Read the official doc about Dependency Injection for some of the motivation behind its use.

Friday, May 24, 2013

Angular service or factory?

tl;dr is at the end

In various AngularJS tutorials and documentation, the authors choose to use service or factory but don't explain why you would use one or the other. Few mention that value and constant are also options.

Let's see why you would use one over the other. We should also understand how providers work:

provider

Here's the source for the provider method:

function provider(name, provider_) {
if (isFunction(provider_) || isArray(provider_)) {
provider_ = providerInjector.instantiate(provider_);
}
if (!provider_.$get) {
throw Error('Provider ' + name + ' must define $get factory method.');
}
return providerCache[name + providerSuffix] = provider_;
}

name is a string. provider_ can be one of three things:

  1. function

    If a function is passed in, the function is called with dependency injection and should return an object with a $get method.

  2. array

    An array will be treated like a function using Inline Annotation. It must also return an object with a $get method.

  3. object

    If an object is passed in, it is simply expected to have a $get method.

Whatever the second arg to provider is, you eventually end up with an object that has a $get method. Here's an example showing what happens:

// You can run this

// Create a module
var hippo = angular.module('hippo', []);

// Register an object provider
hippo.provider('awesome', {
$get: function() {
return 'awesome data';
}
});

// Get the injector (this happens behind the scenes in angular apps)
var injector = angular.injector(['hippo', 'ng']);

// Call a function with dependency injection
injector.invoke(function(awesome) {
console.log('awesome == ' + awesome);
});

Once you understand providers you will see that factory, service, value and constant are just convenience methods for making providers.

factory

Here's the source:

function factory(name, factoryFn) {
return provider(name, { $get: factoryFn });
}

So it lets you shorten the awesome provider creation code to this:

hippo.factory('awesome', function() {
return 'awesome data';
})

service

Here's the source:

function service(name, constructor) {
return factory(name, ['$injector', function($injector) {
return $injector.instantiate(constructor);
}]);
}

So it lets you make a factory that will instantiate a "class". For example:

var gandalf = angular.module('gandalf', []);

function Gandalf() {
this.color = 'grey';
}
Gandalf.prototype.comeBack = function() {
this.color = 'white';
}

gandalf.service('gandalfService', Gandalf);

var injector = angular.injector(['gandalf', 'ng']);

injector.invoke(function(gandalfService) {
console.log(gandalfService.color);
gandalfService.comeBack()
console.log(gandalfService.color);
});

The above code will instantiate Gandalf, but remember that everything that uses the service will get the same instance! (which is a good thing).

value

Here's the source:

function value(name, value) {
return factory(name, valueFn(value));
}

Using value would let you shorten the awesome provider to:

hippo.value('awesome', 'awesome data');

constant

Here's the source

function constant(name, value) {
providerCache[name] = value;
instanceCache[name] = value;
}

constant differs from value in that it's accessible during config. Here's how you use it:

var joe = angular.module('joe', []);

joe.constant('bobTheConstant', 'a value');
joe.value('samTheValue', 'a different value');

joe.config(function(bobTheConstant) {
console.log(bobTheConstant);
});

joe.config(function(samTheValue) {
console.log(samTheValue);
});

// This will fail with "Error: Unknown provider: samTheValue from joe"
var injector = angular.injector(['joe', 'ng']);

Read Module Loading & Dependencies in the Modules doc for more information on usage.

In summary

If you want your function to be called like a normal function, use factory. If you want your function to be instantiated with the new operator, use service. If you don't know the difference, use factory.

This is the (great) documentation for each function in the AngularJS source:

  1. factory

    A short hand for configuring services if only `$get` method is required.

  2. service

    A short hand for registering service of given class.

  3. value

    A short hand for configuring services if the `$get` method is a constant.

  4. constant

    A constant value, but unlike {@link AUTO.$provide#value value} it can be injected into configuration function (other modules) and it is not interceptable by {@link AUTO.$provide#decorator decorator}.

Wednesday, May 22, 2013

Angular and components

I've been playing around with AngularJS lately.  It's okay.  It's very weird.  They've named some things poorly (and magically), the documentation is not great and there are some unexpected quirks.  I feel like AngularJS is the first step toward a new class of JavaScript libraries, but it's not the end.

Something is going to come along and choose good names (without magic '@', '=', '&') with great documentation.  Just today, I happened on this talk:



Which does a lot of AngularJS things.

I haven't explained thoroughly... sorry.  Maybe when I have more time, I'll post more.

Friday, May 10, 2013

@#$%!


To all you producers of words: please stop swearing.  For the sake of your audience, elevate your language above crude speech.  You lose respect when you swear.


You're not dying!

I once watched a video about API design.  Rather, I once watched part of a video about API design.  I had to stop after the 5th or 6th expletive.

API design.

API design?!  A discussion about API design is probably the last thing in the world that deserves to be littered with swearing!

You're not dying.  Your family isn't dying.  Your arm isn't being severed.  And even these terrible situations don't merit debasing yourself by swearing -- much less API design.


I really want(ed) to share your stuff...

Too frequently, I'm reading an interesting article when @#$&!  Out of nowhere (and completely out of context) I'm jarred by a four-letter word.  Why did they ruin it?  I wanted to share this and now I don't.

I don't want to share with my coworkers, family members or friends something profane -- for sharing implies endorsement, and I don't want to be associated with such things.

If you want it to be shareable, leave out the swear words.

Make me laugh, not cringe

Instead of attacking me by swearing, invite me to laugh with you.  Replace poor words with ridiculous or benign ones.  Or channel an unloveable critter/substance/thing to express your shock/displeasure/pain: Rats! Mucus! Taxes!

(Let me be clear: that was "Taxes" not "Texas.")

Yes, swearing sometimes makes some people laugh.  Bathroom humor makes some people laugh, too.  There are much better ways of making people laugh: choose a better way.

And, yes, swearing will garner attention.  A baseball bat to the head will also garner a certain amount of attention.  You don't want that kind of attention.  And in the end, it's better to lead people through charisma and intelligence than through head shots.

Use honey, not vinegar.

You're smarter than that

We don't live in a high school locker room.  Swearing is a cheap alternative to thoughtfulness.  Demonstrate your intelligence by expressing yourself accurately and powerfully through well-chosen words.  People think you're dumb when you swear.  Go ahead, A/B test it.

When your mouth is foul, your brain is discredited by association.

It's who I am

Some will say, "It's who I am.  Deal with it."  Well, they'd probably say, "$#%!, it's who I am *$!@.  @$#& deal with it.  *@&# $%@!"

Swearing isn't an identity.  You control your mouth.  You have a brain.  Use it wisely.

Thank you

To those who speak well, thank you.  I frequent an IRC channel that has an enforced family-friendly policy.  The admins gently remind people to keep it clean, which they generally do.  And it's great!  I feel comfortable there.  I respect the admins and others there.



Keep it clean and make the world better, muskrat!

Tuesday, February 5, 2013

What I wish Puppet and Salt would do differently


I like the idea of configuration management tools like Puppet and Salt; they have promise.  They attempt to solve a huge, complex problem.  I haven't come up with a working solution that is any better.  But if I ever had time to work on a solution it would improve on the current solutions in the following ways:

I wish they would use a standard for state definition


If each tool worked from a standard state description syntax, then people could try different tools without so much more effort.  Perhaps this could be achieved by developing a standard for describing machine state, and then developing tools that convert from the standard state definition to the tool-specific state definition.

A standard would also allow for more general tools to create state definitions.

And a standard would allow interoperability with things that are not configuration management tools: state visualization tools, for instance.


I wish they would stop claiming that you describe state only


Both tools inspire awe when they claim that you only have to describe "the desired state" of resources, rather than a script of things to do.  But they lie!

Puppet manifest files and Salt sls files both include syntax for restarting services, running scripts based on changes or doing other one-time events.  In other words, you are describing both state AND events.

I think the tools could be simplified if they would stop pretending that you can manage a system by describing state alone.


I wish they would focus on getting out of the way instead of cute, colored output

Colored output can be useful.  But when you're coloring output instead of providing useful data, the color is infuriating.

I spent an hour with Salt trying to create a user on a machine.  Every time it failed to create the user, this is the error I would see:

'Failed to create new user lighttpd'

It didn't show the command it was trying to execute.  It didn't show the stdout/stderr from the shell.  The logs didn't help either.  The output was very cute, colorful and useless.  When something runs, I should be able to see the arguments used to spawn the process, the environment variables for the process and the stdin, stdout, stderr and exit code of the process.  Why replace that great debugging information with cute, custom messages and colors?