Systemd Boot order and Tomcat7 and nslcd

I had a problem yesterday with on of our servers not starting up tomcat after a reboot. I eventually tracked it down to an error in the boot ordering. I thought it might be useful to write down the steps I took to work out what was happening and how I fixed it.

First thing is tracking down the error. The logs has the following line:

start-stop-daemon: user 'xyz' not found

Now this suggests that the users I am trying to run tomcat as is not available. This is linked to the fact that the user xyz is a user that comes from ldap. My hunch is that this is a boot order thing. So how do we start to find out what is going on?

Luckily systemd comes with quite a lot of tools to help sort this out. But first of all a quick scan of the logs shows that nslcd, the daemon that provides ldap users, does indeed start after tomcat.

Lets start to look at what is happening. The first tool I picked on is 'systemd-analyse' and this shows a little information.

systemd-analyze dot "tomcat7*"
digraph systemd {
        ""->"tomcat7.service" [color="green"];
        ""->"tomcat7.service" [color="grey66"];
        ""->"tomcat7.service" [color="green"];
        ""->"tomcat7.service" [color="grey66"];
        ""->"tomcat7.service" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"systemd-journald.socket" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"" [color="green"];
        "tomcat7.service"->"system.slice" [color="green"];
        "tomcat7.service"->"" [color="black"];
        "tomcat7.service"->"system.slice" [color="black"];
        "tomcat7.service"->"" [color="grey66"];
        "tomcat7.service"->"" [color="red"];

This is a dot file and you can view it in graphical form using something like dotty but for this small section I can read it fine. We can see here that there is not dependency between the two services that we are interested in. Now it maybe there are other things in play here but lets continue to look at nslcd.

systemd-analyze dot "nslcd*"
   digraph systemd {
        "atd.service"->"nslcd.service" [color="green"];
        "courier-pop-ssl.service"->"nslcd.service" [color="green"];
        "apache2.service"->"nslcd.service" [color="green"];
        "courier-ldap.service"->"nslcd.service" [color="green"];
        "kdm.service"->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="green"];
        "masqmail.service"->"nslcd.service" [color="green"];
        "courier-pop.service"->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="grey66"];
        "kolab-cyrus-common.service"->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="grey66"];
        "nullmailer.service"->"nslcd.service" [color="green"];
        "nslcd.service"->"system.slice" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"slapd.service" [color="green"];
        "nslcd.service"->"" [color="green"];
        "nslcd.service"->"systemd-journald.socket" [color="green"];
        "nslcd.service"->"shishi-kdc.service" [color="green"];
        "nslcd.service"->"heimdal-kcm.service" [color="green"];
        "nslcd.service"->"heimdal-kdc.service" [color="green"];
        "nslcd.service"->"krb5-kdc.service" [color="green"];
        "nslcd.service"->"systemd-journald-dev-log.socket" [color="green"];
        "nslcd.service"->"" [color="black"];
        "nslcd.service"->"system.slice" [color="black"];
        "nslcd.service"->"" [color="grey66"];
        "nslcd.service"->"" [color="red"];
        "citadel.service"->"nslcd.service" [color="green"];
        "courier-mta.service"->"nslcd.service" [color="green"];
        "cyrus-imapd.service"->"nslcd.service" [color="green"];
        "sendmail.service"->"nslcd.service" [color="green"];
        "cron.service"->"nslcd.service" [color="green"];
        "wdm.service"->"nslcd.service" [color="green"];
        "xdm.service"->"nslcd.service" [color="green"];
        "courier-mta-ssl.service"->"nslcd.service" [color="green"];
        "am-utils.service"->"nslcd.service" [color="green"];
        "slim.service"->"nslcd.service" [color="green"];
        "autofs.service"->"nslcd.service" [color="green"];
        ""->"nslcd.service" [color="green"];
        "display-manager.service"->"nslcd.service" [color="green"];
        "gdm3.service"->"nslcd.service" [color="green"];
        "exim4.service"->"nslcd.service" [color="green"];
        "dovecot.service"->"nslcd.service" [color="green"];

Again not link between the two but notice all those other services? I think we are heading in the right direction. Time for a different tool now. Lets look at the config for some of these services.

systemctl cat tomcat7
    # /run/systemd/generator.late/tomcat7.service
    # Automatically generated by systemd-sysv-generator

    Description=LSB: Start Tomcat.

    ExecStart=/etc/init.d/tomcat7 start
    ExecStop=/etc/init.d/tomcat7 stop

This tells us a couple of things. First off systemd is using the old sysv init script to start tomcat. And second there is little in there to indicate a dependency on anything more than a basic system. Now lets look at nslcd

systemctl cat nslcd
    # Automatically generated by systemd-sysv-generator

    Description=LSB: LDAP connection daemon


Ah okay antother sysv init script and this is where my deps are defined. Not that this info is different from the systemd-analyse output as it shows the config from the files and not a full dependency tree. So lets have a look at the top of the nslcd init script:

head -35 /etc/init.d/nslcd |tail -15

# Provides:          nslcd
# Required-Start:    $remote_fs $syslog $time
# Required-Stop:     $remote_fs $syslog
# Should-Start:      $named $network slapd krb5-kdc heimdal-kdc heimdal-kcm shishi-kdc
# Should-Stop:       $network
# X-Start-Before:    $mail-transport-agent $x-display-manager am-utils apache2 atd autofs citadel courier-ldap courier-mta courier-mta-ssl courier-pop courier-pop-ssl cron cyrus-imapd dovecot exim4 gdm3 kdm kolab-cyrus-common mail-transport-agent masqmail nullmailer sendmail slim wdm xdm
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: LDAP connection daemon
# Description:       nslcd is a LDAP connection daemon that is used to
#                    do LDAP queries for the NSS and PAM modules.

These are the LSB headers that systemd will use to work out the start order for old sysv init scripts. And the X-Start-Before header is the one we want to fix. Adding "tomcat7 tomcat8" to the end of that line will "fix" this and I have added a bug to ubuntu to try and get that fixed.

In actual fact I ended up adding "nslcd" to the end of the Required-Start line in the tomcat7 init script as it felt like a better fit for my setup.

I think the correct systemd way is to add a file '/etc/systemd/system/tomcat7.service.d/override.conf' with the following content


This can be done simply by running 'systemctl edit tomcat7'.

Either of those solution seem to be a little odd, having all those services hard coded seems odd. Maybe a better solution would be to have another $ service such as $networkusers and have services depend on that?

I made myself some bathroom drawers.

Bathroom Drawers

Over the last few weeks I have made a set of bathroom drawers. I had one of those wire racks that came with the flat I am renting and its just a mess, slightly rusty and collecting dust. Plus it does not fit everything in.

I set about making a simple set of drawers. I was not going from any designs so just made it square all round. I wanted the bottom drawer to hinge out at the bottom to give access in the small space that it fits in. The bottom drawer is all front as it takes mostly tall bottles of cleaner and the like. Seems to work quite well. Top drawer is quite standard I think. I brought the back in a little so that the side rails are still supporting it when you can see everything in it.

Top Drawer Until I had finished it there was still no plan for the finish. I quite like a shellac or Osmo oil but I don't actually like pine furniture that much so I went with a grey paint. It's a Rustins chalk paint so it went on really easy. I put a coat of shellac on the whole thing inside and out before I had fitted the bottom drawer. Then the chalk paint, two coats. Finished the whole thing off with a couple of coats of Osmo Oil.

This project to a little longer than expected I think mostly because I was not confident about the bottom hinge idea so kept putting off building it up. It turns out this was very simple in the end and the system seems to work very well so far.

Bottom Drawer It's another wonky build, the main frame just came out slightly off square all round so I had to shave the bottom flat and make the drawers to fit. I think I got frustrated at that point so did not finish the joints or sides off properly and it shows on the final object. One of the differences with a paint finish is that the grain pattern does not cover up any of the badly finished bits. It's a shame looking back that I did not go over those bits with more care. All told I think I know the exact times that I was doing the marking out and was tired so not paying enough attention. Layout and detail is critical so I should probably save them for a good day.

It's all a bit chunky on the inside. Especially the top drawer which I made out of the same wood I used for the main body of the box. I should have used thinner stuff. Same for the bottom one. Front bits are okay, just the sides and back. It won't wear out I suppose.

Oh and that big knot on the front is rather annoying. Again pay attention when marking up.

Europe is not about money and laws it's about people.

The debate about Europe is not one of how much we pay, or who is allowed to live where, or what laws are made. Europe is about breaking down borders and living with your neighbour but nobody will mention that in the next few months.

The Tories have come back from Europe with some tweaks to the treaty most of which are small minded and self centred. But at least we have got that bit over with and can get on with the real debate. Or at least that is what you would hope but both sides have already started to spout rubbish and fight over insignificant details.

David Cameron thinks that the big issues are things like protecting the banks from international regulation. It's like he just slept through the last recession. He wants to stop people coming to this country claiming benefits. A policy that looks more aimed at the UKIP voters and middle England who think that it's really a problem, than at making any positive change. He is haggling over the lives of people struggling to make their way in the world to save a few million on the budget while billions go missing under laws written to allow large companies to move money out of the country. He wants to protect the City because he can't see anything better about this country than the cash it generates in the City. He talks about "British Values" then wants to stop them at our borders.

The way to tackle migration is to make it less attractive for people to leave their home countries. (Assuming you think it needs tackling) If you want to stop people from Poland coming to the UK then invest in Poland. Then we can get on with letting people come and go as they please. Similar to my idea on how York should solve it's house price issues by investing in neighbouring Selby. Europe is set up to help with this. The flow of labour allows people not only to work where they want but also encourages people to spend time outside of their borders and experience life in other peoples shoes. The flow of Money allows areas that are less attractive to get a start on the ladder. When you look at the difference European money has made to places like Sheffield giving it the breathing space to rebuild and redefine itself. I hope the same would be done in Donetsk it's twin city in the Ukraine (incidentally a city founded by a Welsh immigrant) Allowing that city to grow in the same way. Supporting everybody through tough times. (Obviously not now as Russia annexed it).

At the moment we think we are in a bad way but the solution is not to go back to small states fighting each other. Let not lash out at groups of people that are a little different and blame them for all our problems. Europe has tried that before and it turned out rather bad.

Europe is much more than laws and budgets its about a sense of belonging, of freedom, of being part of one thing and not lots of little things fighting each other. Lets see if this is the focus of the coming months.

Made some cable ties

Inspired by a youtube video from Mod in a Box about creating cable ties from wire I finally thought of a use for that half meter of 50A mains cable I had lying around.


So taking the wire, some Sugru and a pair of pliers I set to work.


I stripped off the outer grey covering and cut the wire into some random lengths. Anywhere from about 5cm to 15cm. Even really short ones seem to work as they rely on the bend in the wire more than the twisting.


Next up I bent the end of the wire back on itself. Turns out nice big pliers are the best for this not pointy ones. The wire is very stiff. Put a little paper or card over the grip on the pliers so you don't mark the outer shielding.


Next up put a small blob of Sugru on the end. Just enough to seal it up and cover any damaged bits. You only really want to stop it catching on things and make it look smart.


Then just leave them to dry for 24hrs. Then you have some really nice cable ties. I like how the close round any cable and hold it together in a small clump without having those horrible pointy wire ends.

Evening classes and my new Hi-Fi stand

There will be an exhibition of my work in the York Explore Library this weekend ( 20-21 June 2015 ). Well technically it's not about my work but the York Adult Learning exhibition but hey, I have to start somewhere and I have one thing on show!


It started when I signed up of an evening class with York Council ( I think this is the website but I can't make it's search work ) it was called Furniture Design, Make or Restore. With Julian Marston.

Basically every Monday night I would cycle over to the school where it was held and get some woodwork done.


First day was a bit daunting as you would exact, everything new. I was late of course ( I may have stopped for some chips ). But did not miss much of the introduction. There seemed to be a few people who where regulars and had just started to get on with things and a few like myself who had no idea what it was all about. Julian gave us a quick tour round all the tools that we might want to use and then let us have a go on them. Band saw, pillar drill and belt sander. Then just left us to it. I was later to realise that was generally his style. If you needed help he would be there with advice and help but other than a chat now and again you where free to get on with things.


That first day I chatted with some of the people there wandered around a bit, and had some tea. Then went away with the puzzling question about what to make. It's not like I don't have enough things I want to do but this had to fit in ten weeks and be able to go back and forth on my bike each week. I pondered for a bit then chose to make a stand for my Hi-Fi. I have recently sold all my large Hi-Fi. Gone is the large surround amp, separate DAB radio and CD player and now I just have a small amp and a squeezebox. So I wanted a simpler stand that reflected that. My speakers have also shrunk but have matching stands that look rather neat so decided to mimic that in wood.

The design phase consisted of a couple of sheets of paper and a very rough sketch. Mostly I was going to wing it as I don't really know what I am doing. Measuring things up I headed out to the local timber merchant. I came back with a couple of planks of oak. This was going to be a fancy stand after all. For reference a couple of 2m oak planks is no problem on the bike.

A little bit of planning now had to happen as I wanted to glue the planks to make a sheet. So I did that at home. This process, as with most of mine at the moment, started with an hour watching YouTube. After realising that I needed an electric jointer, planer and table saw I managed to find some good videos showing what I was supposed to be doing. Paul Sellers and The English Woodworker where a great help as always.

So I took my newly created sheets of wood to school and tried to plane them flat. I had not realised how good it was not to have to chase the workbench around the room as you plane. Even if these benches where a little on the low side. Planing went well and I am still surprised how smooth and neat it all looks with just a plane much better than an electric sander. The second sheet seemed a lot harder, no matter how I tried it just seem to either dig in or skim over. It was at that point that Julian popped over and causally mentioned that he had a sharpening stone if I needed it. How right he was! Just a couple of seconds on the stone and I was back in business. I let Julian show me how so I could pick up some more sharpening tips. I have some diamond stones at home and at the time was struggling to get things right with them so I guessed I would be no better with the oil stone.


A scratch block is a really old style tool consisting of something metal jammed into a bit of wood. You then drag this down the side of your work piece and it scores a line in it. I made one from a screw with the edge filed off. It worked really well and was a lot less scary to used than a router. With this I scored the three lines down the front to reflect the same pattern in my speaker stands.


Next up was the glass. York Glass Supplies where really great. I took in my paper template, drawn up in FreeCAD, and they just cut it out and drilled the holes. Even smoothed over the edges nicely. There was a slight delay while they waited for the right sized drill bit to be delivered from Germany but other than that a really great service.

( Funny story about the York glass supplies website. Their ssl certificate shows as bad because it is only valid for I decided not to look any further into that )


Next up I had a go on the lathe. What fun. Sawdust everywhere! I had a practice making a square block round then went ahead an made up the little feed to go on the end of the glass and the front.

Then there was the rounding of the edges. Marked up with pencil I went at it with the plane. First at 45 degrees then gradually smoothing that out. Then a little tidy up with a rhasp and things where starting to look tidy.


I had to do all my gluing up at home so that it would have time to dry before going on the bike. It worked out quite well as I had just treated myself to a load of clamps. We will see if it all holds together in the long run but I don't think there where many gaps which is the important thing.

Once I had the mortice and tenon cut for the top it was time for the finish to go on. This was however after the end of the course so none of my workmates got to see the final showing. I used some Osmo oil partly because I like the mat finish and partly because it's really easy to apply with just a scouring cloth.

All done and in place. I am really please with how it turned out. I am sure I would do things a little differently next time round but I think it looks pretty neat.


Starting nfc-eventd from systemd on the beaglebone

Attaching an nfc reader to the beaglebone. In my case a cheap one called COOQROBOT which I think is pn532 variant.

I installed both libnfc and nfc-eventd from source. The ususal ./configure; make ; sudo make install. ( You might need to follow this with a ldconfig -v to get the library picked up )

Now you can start a service that monitors your nfc reader for new tags. Just drop the following into /etc/systemd/system/nfceventd.service and run sudo systemd daemon-reload.




Now you can start the service with systemctl start nfceventd.service. And look at errors and output with systemctl status nfceventd.service.

By default the config for nfc-eventd just saves the events to /tmp/nfc-eventd. If you edit /usr/local/etc/nfc-eventd.conf you can change what this now does. So it could for instance send those events to your node-red server via mqtt.

action = "mosquitto_pub -h -t /nfc/tag/ -m $TAG_UID "

Bash Linting. Checking your scripts are nice and lint free

Continiuing on from my previous post on testing shell scripts with bats I have been looking for a linting solution for shell scripts. And I think I have found a nice one in shellcheck. Which is aviailble as both a website and a packge on debian.

It's really simple to run just give it your shell script as an argument and you get a nice set of comments on the style of your code. Here is the result of testing their test script.

#!/usr/bin/env bash
## Example of a broken script. Hit the Down Arrow button to ShellCheck it!
for f in $(ls *.m3u)
  grep -qi hq.*mp3 $f \
    && echo 'Playlist $f contains a HQ file in mp3 format'

Then run shellcheck

In line 3:
for f in $(ls *.m3u)
         ^-- SC2045: Iterating over ls output is fragile. Use globs.
              ^-- SC2035: Use ./*.m3u so names with dashes won't become options.

In line 5:
  grep -qi hq.*mp3 $f \
           ^-- SC2062: Quote the grep pattern so the shell won't interpret it.
                   ^-- SC2086: Double quote to prevent globbing and word splitting.

In line 6:
    && echo 'Playlist $f contains a HQ file in mp3 format'
            ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

I have a simple makefile now that's just there mostly as a reminder of the testing that is aviailble for that script.

bats: *.bats
    bats *.bats

lint: your_script your_nextscript
    shellcheck your_script
    shellcheck your_nextscript

Next up is intergration with vim. I use the really nice syntastic vim plugin. Syntastic does automatic style checking on files as you save them and shows a set of errors. Installing it is simple. Install shellcheck, I use apt-get install shellcheck on Debian but you can use your favorite way on your OS. Then install the vim syntastic, again I use Vundle so adding Bundle scrooloose/syntastic to my .vimrc and then running BundleInstall in vim did the trick for me. Now when I write a shell script to disk vim will comment on my style.

Setting Soundcard names in Gnome

I have a couple of sound cards attached to my laptop at work and it gets confusing to know which one is which when they have names like "Analogue line out". So I set about changing them and struggled a bit in the process so I thought I would note it here for reference.

Gnome takes the name of the soundcards from pulseaudio. There does not seem to be a way to rename them directly in gnome but you can do it from the command line. The command pacmd list-sinks will give you a list of the devices that you have. Looking through that list at the 'name:' field are the identifiers that you can use. ( You can also use the ID but I am not sure how stable they are ) The man page for pulse-cli-syntax is the rather confusing location for the commands that you can give the pacmd command.

Now we know which card we want we can change the description property that Gnome uses with the following command. Note the extra quotes are required if your description has spaces in it.

update-sink-proplist alsa_output.usb-Burr-Brown_from_TI_USB_Audio_DAC-00-DAC.analog-stereo 'device.description="External Blue headphones"'

Then you can run the pacmd list-sinks command to check that works. At the gnome level nothing will have changed. So now you need to add this to your config file ~/.config/pulse/ which is just a list of pacmd commands to run on startup. You can now drop the extra single quotes as it does not need to be escaped for the shell prompt.

Update: You need to add the line below to your so that it knows about the default config. Pulseaudio only loads one config file by default so this just makes sure you get the default settings first.

.include /etc/pulse/

Now it just remains to get pulseaudio to re-read it's config in the usual unix way by giving it a HUP.

pkill -HUP pulseaudio

Listening to SSL with socat

I wanted to dump the headers for a http request over ssl today. Pulled out socat and this command and it seemed to work quite nicely.

socat OPENSSL-LISTEN:4433,cert=server.crt,key=server.key,verify=0 -

You will need to create server.crt and server.key and if you don't want the other end to complain then they should be a valid keypair.

Socat is a really nice tool that exposes the power of Unix sockets and allows you to connect anything together. In this case a tcp socket with ssl support and the standard out. But it could be a UDP port and a serial cable for all all socat cares. A really nice tool.

Testing Bash with Bats

Spectacled Fruit Bat by Shek Graham

Spectacled Fruit Bat by Shek Graham

I quite like unit tests. They give you that quick feedback that things are going to be okay. I can write a test watch it fail. Write some code and watch it fail also. Then learn very quickly what I have done wrong.

At some point last year I read the Clean Code book by Rober C Martin and it opened up to me a lot of the ideas about testing and code that I had not really got to grips with at the time. Martin takes you through the reasoning and process for producing code that, in a years time, you might want to actually look at and feel comfortable editing.

What has this go to do with bash and bats? Well we have been adding more tests to our puppet code and it came time to write a bit of bash for moving some files around. I thought there must be a simple way to check some of this and sure enought that is where bats comes in.

bats is a testing framework for bash scripts. I know it sounds crazy but it really does work. If you start applying some of the lessons from clean code and spliting your work into functions with local varaiables it soon becomes easier to test. You can, as ever, take things too far but this does at least allow you to test a lot of your logic before you roll it out. With the added benifit of makeing the code a little more readable.

I found this blog post by Spike cleared up some of the questions I had about testing functions as well.

Some other useful things are the google shell style guide and shUnit2 which I have yet to look at. The Defensive Shell Guide by Kfir Lavi and Better Bash Scripting in 15 Minutes by Robert Muth are also good.