Over the past four years we’ve deployed a puppet/foreman environment to support Ubuntu 12.04 and 14.04 for our research and production Linux systems. As 12 is approaching end of life and there are no longer Foreman updates available we decided it was a good time to revisit our overall puppet/foreman integration. Over the years it had slowly grown to include a bit of cruft and needed a good haircut. In addition, during the past four years, Foreman had added additional features which made it a good time revisit how the two communicate and where the hand-off in responsibility lie. So with that introduction, the environment we deployed had the following goals and challenges in mind:
- Clearly define the hand-off between Foreman and Puppet.
- Profiles and roles vs smart parameters vs host groups vs config groups.
- Create a clear development to production workflow.
- Module updating, development, testing.
- How do we support operational testing (ie, patching, etc) vs longer term development.
- Improve git integration (yeah, part of above, but a major motivation in of itself)
- How can we support multiple development efforts?
Final Environment
Skipping ahead to the end, our final Foreman environment consists of the following. We ended up creating a separate dev and production environment which were 100% independent, yet mirrors of each other. We developed a workflow for allowing each developer to have their own environment and setup a clean separation between this development environment and the testing environment/production environment. This allows the production side to quickly test and apply security and other upstream updates without impacting longer term development efforts.
Foreman and Puppet
- Two Foreman instances, dev and production, each on its own subnet.
- Only profile classes and their parameters are exposed to foreman.
- Configuration groups within foreman are used to create roles.
- Multiple base host groups, one for each subnet/authentication domain.
- Base host group handles core auth/patching.
- Second level host groups create services (ie, HPC Cluster Node) and have configuration groups applied, and contain all hosts.
- Hosts/services are created in pairs, a patch-testing and production both on the production environment.
- Each production has a corresponding test host which mirrors production and is used to test patches and other minor updates.
- The testing host also serves as a recovery point, meaning that a nightly backup can quickly be applied and it can be moved into a production role.
- Development VM’s are attached to the dev foreman server and are short lived hosts for developing and debugging puppet configurations.
Git and Environments
- Single git repo for puppet all puppet modules and profiles.
- Git branches mostly correspond to puppet environments
- ie: development == /etc/code/pupplabs/code/environments/development
- production – production environment on production foreman server
- development – set as ‘production’ on development foreman server, development on production foreman.
- Each developer/sysadmin has their own environment on the dev server, within their environment they can switch to the appropriate branch or use their own developer forks
- Code workflow:
- Developer or sysadmin works in own branch.
- Pull request to development branch and import on dev server.
- Review and testing in development (if substantial chages, additional step of verifying on production server development environment)
- Pull request to production and set parameters.
In parts two and three, I’ll cover a bit more about the motivation and detail behind this setup.
Some additional reading:
- Google groups: “Foreman and Puppet – What does what?“