I discovered the puppet define command the other day and thought it would be useful to be able to access that from within vim so I created a little function to do it. Thought it might be useful for others so I have noted it here.

Just drop this into ~/.vim/ftplugin/puppet.vim and you should be good to go. Typing \pt ( ,puppet,type ) when over a puppet type should pop up a definition.

Edit: Now available to download from github.

" Takes some settings
" g:puppet_command The location of the puppet command
if !exists('g:puppet_command')
  let g:puppet_command = 'puppet'
endif
" g:puppet_doc_widown - The type of puppet doc window to open
if !exists('g:puppet_doc_window')
  let g:puppet_doc_window = 'split'
endif

"

" Define a puppet type
function ?PuppetDefineType(fname)
  " Show the puppet definition of the current puppet type that the cursor is
  " sitting on.

  " Create a window with a new __doc__ buffer
  if bufnr("__doc__") >0
      execute g:puppet_doc_window.'| b __doc__'
  else
      execute g:puppet_doc_window.' __doc__'
  endif
  " Make this buffer disposable
  setlocal noswapfile
  set buftype=nofile
  setlocal modifiable
  " Cleare the buffer
  normal ggdG
  "Read in the new description
  execute 'silent read !'.g:puppet_command.' describe '.fname
  " Go back to the top
  normal 1G
  setlocal nomodified
  set filetype=rst

endfunction

" Setup two commands to define the word under the cursor
map <buffer> <leader>pt :call ?PuppetDefineType('<C-R><C-W>')<CR>
map <buffer> <leader>pT :call ?PuppetDefineType('<C-R><C-A>')<CR>

Things that may improve this are the ability to run the command from anywhere in the file and have it find the nearest type and referencing local types. Let me know if you find it useful and I may ask for it to be added to Rodjek's puppet vim module

We are planning some software upgrades to our main website and as in previous iterations I started to write some tests. Just some basic stuff to make so that the config that we have set up does what we think it should do. I am not testing the applications or content yet but hopefully some of what we have learnt can help others to do that.

In the past I have written some unit tests in python that where simple and ran from the command line. This time we thought that it would be good to make these test slightly more permanent.

So I picked some tools as detailed below and we set about working out a set of tests. For a start we have just trawled through the long and historical Apache config picking out bits that look important and just writing a feature header. Now we are going back and putting in the details of all the tests. This is a great start and it's almost fun to see the graph of successful tests go slowly blue. (Jenkins seems to like blue not green for test results but there is a plugin to change that if you want. There is a plugin for everything!)

The aim of creating these test is obviously to give more confidence when we move over to the new server but I also hope that we can move to a different way of managing configuration changes. Moving to a state where we write the test first and then the config should be a much more relaxed situation. There is also the ability to raise alerts into Nagios if we require it. So if we see some security tests fail we could raise a Nagios alert and quickly take action.

Writing tests has also forced us to go back through the old configs and ask some questions about what each section is trying to achieve. I have already removed one section that is no longer relevant and have earmarked a couple more for after the upgrade.

We are still adding tests to the features that we defined in the first run through and that will give us a good base to be going on with but I am sure that there are more that we can add that are not directly related to the Apache config. Testing all the redirect rules is a massive job and we will need to automate it at least for the initial set up. When we have something that is useful I would like to hand the management over to the web authors for the most part.

Next up is to run puppet-lint over the puppet config from jenkins and shame us all into fixing our style and then to start writing some puppet-cucumber test! I already have it sending test results to our IM server for maximum annoyance"

Tools

I will go through some of the technical bits of the tools for those that are not familiar with them but there is better information on their respective websites.

Cucumber

Cucumber is a BDD[Build Driven Development] tool that allows you to write feature definitions in English and then parse them to create real test. You right out a feature and then some scenarios that would test a feature in a form of English, or your local language, called Gherkin. The resulting file looks a bit like this

Feature: The web server should have a standard set of error pages

  Scenario: Requesting a non-existent page should give the code 404 and the default page
    When I visit /notarealpage.html
    Then I should get a response code 404
    And the page should contain the content "University of York"

The first word of each of those lines is a keyword. Scenario, When, Then, And are all used by Gherkin to know how to interpret the line. You then write some ruby code that can deal with each of the lines. So for the When line you may write.

When /I visit (.*)/ |page|
  @url = "http://#{@host}/#{page}"
  @response = fetchpage(@url)
end

Then /I should get a response code (\d+)/ |responsecode|
  @response.code.should == responsecode
end

Cucumber adds some extra keywords to work with gherkin templates but basically it's just ruby. The first part builds a full url using the '@host' variable that we have set previously in the environment and the page that comes from Gerkin line. The second part matches the response code check and use RSpec's should feature to test it. I am assuming that we have already defined the 'fetchpage' function.

As you can probably see now I can write easily read tests in English for all my error pages without having to add any more code.

These are some really basic examples and I would advise you to go and read the site at http://cuckes.info/ to get a real feel for the full features of cucumber.

Jenkins

Jenkins is a fork of Hudson. Hudson is a fork of Jenkins. They are both a Continuous Integration(CI) servers and separated by politics. Jenkins seems to be the most active and supported. CI is a workflow that means as code is checked into a repository, it is built and tested automatically and the reports, from that code change, are alerted on and stored. I am just using it for the nice web gui and its support for running test regularly and interpreting the results. It has plugins for supporting all types of test and builds from Maven and ant scripts to test coverage, parser warnings and lint packages like jslint or puppet-lint. Again go and read more at http://jenkins-ci.org/. Running cucumber with the command '''cucumber -format junit --out logs/''' will create a load of junit format xml files that jenkins can understand and show you pretty graphs, and details of errors.

This morning Squidward decided that he was going to move out from Bikini Bottom and away from this madness so he phoned the estate agent and she said it would be no problem to sell his house as long as his neighbours where okay. "Oh dear" thought Squidward. So he decided on a plan; to tell Spongebob that it was opposite day so he was going to act crazy all day and Spongebob should play the straight man. It worked for a bit but as usual Spongebob had to take things to far and when the estate agent turned up hilarity ensued. Suffice to say that Squidward will never leave Bikini Bottom. I then left for the conference.

Last night was the conference dinner. Google had subsidised some of the meal which is nice of them and we where booked into the Cave, under South Bridge with the cows bottom sticking out the wall. Meal was good and the talk by Charles Yarnold was great. I think he has one of those geek dream jobs hacking crazy inventions for Gadget Geeks on SKY.

First up this morning was Simon Phipps talking about how the OSI is adjusting itself to the changing climate of opensource. They are moving from a board of 10 members to more of an open organisation with many members. Currently taking in open projects like Plone, KDE, Mozilla, Apache, Creative Commons and Debian but also hoping to accept personal and corporate memberships in the near future. Also starting to act a guiding voice for business and government to try and skew the conversation back from the corporations that are trying to preserve their older business models at all costs.

http://openrightsgroup.org/ - They are one of the few voices that are speaking to government in defence of public rights. If you think their language is too strong then you can only imagine what is being said behind closed doors on the other side of the argument.

Tariq Rashid gave a great talk about all the work that is being done inside government to level the playing field when it comes to opensource and standards inside government. They are laying down the law, quite literally in some cases, about how open software should be considered for all purchases. Whilst his language was obviously designed to appeal to political and corporate interests I can't help feel that the constant focus on the free as in beer angle is not all that needs to be stressed. I think he has a massive job though so whatever works I suppose.

A quick break and some more fluids and it was back in to a talk on OpenAFS. I always knew it was cool and even had a go at setting it up once but I think it's a large investment. Interesting to see that he said that the move to git for version control has really upped the involvement levels and what is quite an old distributed filesystem is looking very suitable for modern requirements. They are adding mobile clients and a web interface is working right now.

A small violin performance before a talk on ldap performance from Howard Chu. Always impressed with people who can actually play and in front of an audience. The improvements to openLDAP by ditching all their caching layers and just going for a memory mapped storage database where impressive not only from the massive improvements in speed but also ditching a lot of the need for tuning. Reminded me of some of the stuff I was reading about Varnish cache. Looks like that it the LDAP server to look at. He also said that he ported the database backend to sqlite among other things and it made a huge improvement there too.

After lunch and some time in the sun it was a talk about Rudder which is another config management tool but this time it's all about simplicity of use. There are still configs that you apply to nodes but this config can be done in a nice web front end and the software builds CFEngine templates that are automatically pushed to the correct clients. It is quite a complete system allowing with packages for the clients logging and monitoring all built in.

Another configuration talk now and I must admit I was flagging a bit now which was a shame because this was quite an interesting concept. We currently provision servers by stating a configuration that is supposed to be running on them but with no concept of any sort state. I can change a server to turn off a service on one side and then bring it up somewhere else but if I want to manage the loadbalancer to make sure that the service is never interrupted then I have to do that manually. Herry Herry's concept was to add some rules that allow you to say things like: Don't turn off that service until the other one is up and the loadbalancer has switched over. He was concentrating on creating a language to encapsulate your rules and a compiler to create a set of steps required to make things work in the right order. It is just a test tool right now but I can see it being a great addition to puppet.

Then the Ligtning talks, just 5 mins so just one line...

Tech Talk PSE - HTML based presentation tool, that can include shell.

pairvm - drbd based server centre fail over for kvm. ( might want to look at ganeti)

time travel for linux - systemTap script to lie to the application about the date being 2038 to test what happens at the end of a pension.

cisco firewall issue - SAC packets id rewritten by firewall - tcpdump rewrites the numbers again.

Machination app store for machine config: users can select apps for their servers.

postgresql update - power management DB sleep when idle and performance is much faster. 9.2 in september.

virtulization concidred harmfull. - we don't need VM's just run Unix processes. Sort of an anti-talk about everything being said!

Samba 4 update - Full AD support is there and now. CIFS clustering. WAN proxy. Preview for release in April.

Your judgment sucks - Humans are buggy as well.

That's it. I am off to the doctor

First day at #flossuk Spring Conference and there is a lot of talk on configuration management, logs, monitoring and devops.

Matthew Bloch of ByteMark gave a great insight into their BigV visualization with lots of interesting details. I think the most surprising thing was that they where using nbd for all their disks. Really simple way to get cheap VM migration and so far really reliable.

Kris Buytaert's 7 Tools of ?DevOps talk gave all the usual CI, monitoring, config management tools but also was keen to stress how important the whole talking and interacting part was. Devops is not about one tool set but more about making sure the communication happens. A lot of the ?DevOps tools are there to help start these conversations. Being able to talk to people about graphs and changes in tangible ways is really useful.

I heard some good reports about the asciidoc talk. Anything that makes documentation easier has to be good. My current thoughts would be to move more of our documentation into the puppet config as Markdown files and api docs. Then run a tool against the repo on check-in to create a web tree.

The talk by Bernd Erk about Icinga, a fork of Nagios, was quite interesting. Nice to see people developing Nagios. They have dome some crazy things like using LDAP to populate the config and adding graphing to the monitors. I think that personally it would make more sense to move this sort of thing into our puppet config but maybe having a node assigned with services in an LDAP tree could be used to provision it with puppet and then also test that service with nagios. As for the graphing I think that feeding those checks out into Graphite is still my preferred option. And I think Patrick Debois has some interesting suggestions for linking tools like graphite, collected, nagios and Ganglia together

Some other small themes that seem to keep cropping up. Ruby is what the cool kids are using. Java is not getting much cheerleading but is used a lot. Chef and puppet seem to be being used or considered by a lot of people.

http://www.flossuk.org/Events/Spring2012/Schedule

Made my joule thief at York Hack Space last night. Most tricky part was winding the coils around the ferrite bead. I was supposed to get 20 turns onto it but I think the beads I brought where a little on the small side. I managed to get 13 or 14 turns on in the end and it works a treat. The final version is running on my desk right now and I have upgraded the bluetak to an elastic band. I will put printing a case for it on my reprap list.

It looks to be only oscillating at 2khz rather than the 50khz I was told to expect but is working great for a battery that was showing less the 1v.

Reading how the circuit works it looks like you could drive any voltage from the coil with a diminishing return in current. I wonder if there would be an optimal number given the components I have.

With the latest version of PrintRun you can now set the slice command. This allows two cool options.

You could set up Slic3r the new cool kid on the block but I want to try using pypy for running Skeinforge. Pypy is a python VM that a really fast JIT so should speed things up.

Setting it up is simple. First download the pypy binary and unpack it somewhere. Then change the 'slicecommand' setting so that, where it did have 'python', it now has '/usr/local/bin/pypy' ( or where you installed pypy ). Now your slicing should be twice as fast. Simple as that.

Seems to be working great for me.

Don't change anything else, still run the 'slicesettings' with standard python as pypy does not have the graphics support out the box and the speed is not reqiored for that part.

My new Seeeduino v3.0 arduino clone arrived today from Cool Components.

As a test I hooked up a DHT11 temp sensor and it all worked a treat. I like the colours too!

So now I know that the office got up to 22 deg C and 49% humidity.

I have added a Arduino page with some details of how I compiled the code without the GUI as at the moment running the GUI crashes my X server and logs me out.

I have been writing some test using cucumber to prepare for a system upgrade of one of our web servers. I am using capybara to do the actual test and have been building up a small set of tests.

Previously I used freshen because I know more python than ruby but there does seem to be a lot more support around cucumber than freshen. I did have a look at the python support in cucumber but everything I read about it seemed to point to it not being great.

Moving on to the point of this post. I was looking at using cucumber-nagios to run some of these tests more regularly on our nagios server. My theory was that I could run all the test regularly and report if any of the parts of the server fail. So testing further into the stack.

The cucumber-nagios website has some info and tutorials to get you started but I soon found that it lacks some of the test that I wanted for things like checking a 404 page etc.

My solution has been to use my existing tests, install cucumber-nagios and just use the output formatting to output to nagios.

cucumber --format Cucumber::Formatter::Nagios features/newserver/ 
CUCUMBER CRITICAL - Critical: 1, Warning: 2, 16 okay | passed=16; failed=1; nosteps=2; total=19; time=2

The last line above is the output of the command and is a format that Nagios can parse and act on. As you can see I still have one test failing. You can then either add the whole test suite into Nagios or pick and choose specific features. You may want to run all the tests occasionally but check some of the status and security features more regularly.

"Thanks Mr Betts". I spent some time putting together an email to my local MP, Clive Betts, a couple of weeks ago. It was prompted by the climate talks in Copenhagen next month. I wanted to voice my concerns that we where going to ignore the future and continue to set vague and distant targets that we can generally ignore. I also suggested that if we spent a little less on building roads and airports or propping up the car industry then we could invest more in real solutions for the worlds growing problems. Suggesting that Britain could once again lead in technology and this time to correct some of the problems we have caused with out previous technologies.

The letter that Mr Betts sent was well considered and seemed to respond to each of my points in turn it. It was not just a circular but a well thought out reply even to the point that he was suggesting local companies that where already doing research into alternative fuels. It is nice to see that he is thinking along the same lines as me and I hope that by adding one more voice he will continue to put the pressure on the government to make the right decisions.

http://www.theyworkforyou.com/ is a great service that really allows you to dig into what your MP is doing for you in government. You can track the questions that they are asking and how they have voted on issues. Looking at Mr Betts' record of voting it looks like this ties in with the words in his letter which is nice to see.

I have been having some really good experiences with Bluetooth and my laptop recently. Fist off I connected my phone by just clicking "Add new Device" from the Bluetooth icon and then selected the "Use this as an Internet Connection" option and it was all setup. Clicked on the network Icon and told it it connect using my phone, it asked me for my network and then connected. No fuss. No driver disk and clunky software. Just connected.

Today I paired up my ?SonyEricsson Bluetooth headphones and again just connect the device from the Bluetooth icon. Clicked on my sound preferences and select the headset for my output and then pressed play in my music player and again it "Just Worked" (TM).

I think that this shows that all the hard work is paying off. All those people who have logged bugs and then those that have fixed them all working to create the correct solution to work across the board. Not just one manufacture bashing things with a hammer until their one piece of hardware works. All the hard work getting drivers fixed to work correctly with pulseaudio is really starting to pay off.