dustycloud/posts/configuration-management-for-the-people.md

title: Configuration Management for the People date: 2014-03-29 14:45 author: Christine Lemmer-Webber tags: configuration, deployment

slug: configuration-management-for-the-people

One of the things I've talked about in the talks I've been giving about MediaGoblin lately is the issue of deployment. Basically it boils down to the following points:

• MediaGoblin is harder than we'd like to deploy, but it's probably easier than most other Python web applications. All modern web applications are currently hard to maintain and deploy.
• We've worked to try to make it so you don't have to know how to be a \$LANGUAGE developer (in this case, Python) to deploy MediaGoblin, but once things break you do have to be a \$LANGUAGE developer, or know one. (We spend a lot of time answering these things in #mediagoblin. This can be improved if we get proper system packaging though (which is currently a work in progress, at least for Debian. I hope to see it soon...))
• Even if you get system packaging, that's not enough. You need to do a mess of other things to set up web applications: configure the web server (Apache, Nginx), configure sending mail, configure a bunch of things. And let's not even talk about getting SSL set up right. Oy.
• The reason people don't see that modern web applications are hard to deploy is because even though they are, there's a team of devops behind the scenes handling it for them.

We can do as good as we can to try to make MediaGoblin's docs easy to understand, but I'm convinced the solution needs to be a layer higher than MediaGoblin. That's probably one of two things:

• A Platform as a Service solution like OpenShift. (Note, there are other proprietary ones out there, but if it's not free software, what's the point?) These solutions are kind of heavy, but they seem like a step in the right direction.
• A configuration and deployment abstraction system, like Puppet/Chef/SaltStack/Ansible, etc.

The latter seems like the right solution to me, but it's not enough as-is. The configuration and deployment systems we presently have are too devops-focused. We can't free society by expecting everyone to join the world of devops... for one thing, these tools are way too hard to learn at present, and for second, a world where only the technically elite are free is a pretty nonfree world, really.

(I don't think things like Docker or virtual machine images are the answer. Aside from being pretty heavy, they don't solve the configuration issue.)

But these systems are pretty close. Close enough even! I think we can pull this off. Let's see what we need and what's missing:

• Needs to have share'able recipes. (Yes yes, people currently do sad, sad things like dumping some recipes to code hosting platforms. That's not the same thing. I want something like an apt repository, but for recipes. (Note, juju might have this... I don't know much about it))
• Recipes should be pretty simple... based on variables set by the user, or variables interpolated by user-specific settings (all these config management systems handle this, I think)... mostly I like what I've seen of how Salt handles this.
• There should be an expectation that you should be able to mix and match recipes somewhat. This means some agreements higher up the chain on how we expect a mail server configuration is going to be described, etc. I'm not sure how the standards/governance around this could be best handled.
• I think a layer on top of all this that's really needed is some kind of web UI for application install and configuration. If I install the MediaGoblin recipe, it may be that a lot of defaults can be set and guessed. But what if I want to turn on the video plugin, and I want to change my authentication system to using LDAP because that's what my company already uses, etc? I think this can be pretty minimal, we can have a specification that both describes config options as well as how to represent them in the web UI.
• It shouldn't be tied to any one specific platform. Not to a wall wart, not to a VPS, not to a Raspberry Pi. It should be generic enough to work on all of these. Again, I see no reason this can't be pretty minimal.

I've been fielding this by people for a while, trying to quiz all the smarter-than-me-people I know about what they think, but I keep coming back to this. At a dinner at LibrePlanet, all the usual suspects were raised as possible solutions, and none of them seemed to fit. I was happy to hear Joey Hess say, "I think that Chris is right, we need the 'layer above apt' configuration solution." (If that wasn't some reassurance that I'm on the right track, I don't know what would be... if anyone would know it would be Joey...)

(Note: GNUToo suggested at FOSDEM that I should look at LuCI... LuCI always felt a bit clumsier than I'd like when I used it but maybe it does do these things. I don't know if it handling recipes like I said, but maybe? Worth looking into.)

Here's a rough plan on how I'd go about doing this, if I had time to do this (which I don't think I do, though I can help someone kick off the project, and I can make some contributions):

• Start by investigating Salt (or Ansible?). It's a little bit heavy but not too heavy (the current python package appears to be 2.8mb...). Plan on using the master-less setup. Salt provides a lot of abstractions around installihng packages and a lot of these other things, so that may be a good start.
• It might be that things need to be paired down to something even simpler, or that Salt is just too hard to build on top of. If so, start simple on recipes. I'd save a dependency graph system as something optional as a "milestone 2" type thing, personally.
• You need some way to bundle a description of the variables to be provided but also how to represent them configuration-wise to the user. I think a Hy DSL might be the fastest way to start writing up variable descriptions including web represenation, but also embed code to retreive results in case a default is not provided.
• At this point you should have a command-line tool that you can run, reads your currently installed recipes and your current settings, and executes them.
• Build a web UI.
• Figure out how to publish up a public repository of recipes. (May need to be distro specific.)

That's it I think. I think if we had something like this, it would simplify deployment greatly.

Are you interested in taking this on? We should talk! ;)

PS: We'd like to do further research and work into making MediaGoblin easier to deploy over the next year. Our capacity to do that largely depends on how much we're able to raise... consider donating to our campaign!

Edit: Mike Linksvayer makes strong points about previous language around PHP; removed. So, reverting language that makes PHP sounds like a problem, but I'll still argue that it's not actually a full solution (there are configuration issues not resolved by language choice)

Edit 2: I guess I didn't say this, so it's worth saying... a lot of difficulties in modern deployment are because people aren't using system packaging (this includes MediaGoblin's docs, which suggest the breaks-all-the-time-even-though-I-understand-why-but-our-users-don't world of Python packaging... we're waiting on Debian packaging. Real Soon Now, I hope!). Using system packaging certainly solves a lot of these headaches, but it doesn't solve nearly all of them. There's still the issues of configuring things, and there's really a lot... too much!... to configure: mail transfer agents, hooking your application up to the web server, SSL, so on and so on. That's something that hasn't been solved, especially not for non-devops people.

I've started cobbling together something that might solve things for non-devops people, and even devops people both! Maybe it'll see the light of day sometime. In the meanwhile, I'm really interested in other peoples' solutions to the problems described above.