Mac OS X Server 5.x El Capitan SSL Certificate Authority

Posted: February 11th, 2016 | Author: | Filed under: Mac OS X, Mac OS X Server, SSL | 3 Comments »

280px-El_Capitan_2005I use Open Directory a lot, I can’t think of many times when I don’t use it at least once a day in some way whether that be direct or indirect. It’s not the best directory system out there – hell, it’s not even very good but it’s what the fine people at Apple have supplied us with and it’s what I use. Although I have long had a pet peeve with the way OD is built if you just run through the setup wizard. The certificate expires in one year. Almost every time I encounter an OD server in the wild nine times out of ten the certs are a mess. They’re either expired or about to expire and all the services that depend on them are freaking out. The clients are constantly being prompted to accept an invalid cert and OD fail-over tends to stop working. My solution has been to build a certificate authority for all my OD installs and to build my own certs that are valid for ten years, that way I won’t have to worry about them expiring and let’s be honest there’s no Mac server on the planet that’s going to last ten years lol (sense the cynicism yet?)

Create the Root Key

First we’re going to hop into a terminal on any Mac OS X box and navigate to somewhere safe in the file system and build the master key, make sure the password you use for this is kept secret and safe.

cd /some/where/safe/I/dont/care/where
mkdir -p myCA/cert myCA/key
cd myCA
openssl genrsa -aes256 -out key/ca.key.pem 4096
chmod 400 key/ca.key.pem

That OpenSSL command is going to ask you a bunch of questions, answer them to the best way you see fit and for the common name put the organization that you’re building this certificate authority for. Do not put an actual domain. Record what you wrote for Common Name because we’re going to need it later. Whitespaces are allowed. Oh and one more pro tip, make sure the value of the certificate common is unique and can’t be half-matched. For example I made a root cert called “Client Name” but when it came to do deploy the certificate via munki, the check install script found a cert called “Client Name Open Directory” and thus the command matched and wouldn’t deploy the new root cert.

Create the Root Cert

openssl req -key key/ca.key.pem -new -x509 -days 3650 -sha256 -extensions v3_ca -out cert/cert.pem
chmod 444 cert/cert.pem

Now we have two files.

cert/cert.pem – This is your CA’s certificate and can be publicly available and of course world readable. You will need to load this certificate into all the clients in your network.

key/ca.key.pem – This is your CA’s private key. Although it is protected with a passphrase you should restrict access to it, so that only root can read it.

Create the First Server key/cert combo

We can now start creating SSL certificates for our various servers and services. Create a directory named a hostname corresponding to the hostname for the computer or service you are creating the certificate for.

Create the first key

openssl genrsa -aes256 -out hostname/hostname.domainname.key 4096

While answering the questions for this make sure you type in the FQDN for server or service you want to secure into Common Name.

Create the server cert

openssl req -new -key hostname/hostname.domainname.key -out hostname/hostname.domainname.csr

Sign the cert

openssl x509 -req -days 3650 -in hostname/hostname.domainname.csr -CA cert/cert.pem -CAkey key/ca.key.pem -set_serial 01 -out hostname/hostname.domainname.crt

Now before we continue I have to drill into you that it’s imperative that you document these certs as you make them, which hosts they’re deployed on to, and what the serial number is. It will help you in the long run.

Finally we’re going to make a passwordless version of the server key, this is the key that we’ll ultimately use on our server. We need a passwordless key so that the Mac OS X services do not need human intervention when trying to use the certs. Otherwise you’ll have to type the cert password in every time you restart the service.

openssl rsa -in hostname/hostname.domainname.key -out hostname/hostname.domainname.key.insecure
mv hostname/hostname.domainname.key hostname/
mv hostname/hostname.domainname.key.insecure hostname/hostname.domainname.key

Import the CA and Cert

Copy the myCA/cert.pem, hostname/hostname.domainname.key, and hostname/hostname.domainname.crt files to your Mac OS X server. Double click on the myCA/cert.pem, this should open the file with Keychain Access which will ask you which keychain to import the cert to. Select the system keychain and then double click on the entry in Keychain Access and set the trust setting to Always Trust.

Next open and click on Certificates. Click on the gear menu and select Import Certificate Identity. Drag in the hostname/hostname.domainname.key, and hostname/hostname.domainname.crt files, when you do this you should see that the files are signed by the organization that you entered in step one. Finally select the cert from the list of certs in and give it time to switch over.

Finally fire up the wiki service in, whip open your browser of choice and connect to the FQDN that you create the SSL cert for. Remember, if your cert common name doesn’t match the way you address the wiki in the URL bar you will get a hostname mismatch error. You MUST make sure the names match from what you type into the certificate’s common name and how you connect to the wiki.

Root Cert Deployment

So now maybe you want to be able to use this cert with some of your clients right? You can do this a multitude of ways. Such as:

Copy the file to /tmp across your network and run the following command as root

security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "/tmp/cert.pem"; srm "/tmp/cert.pem"

Or if you’re cool you’ll use some sort of package deployment system. I use munki because it’s not backed by some money hungry corporation. Here’s looking at you Cohen. 😉 To do this I made a new package with Composer dropped my root cert into /tmp and then wrapped then finished the Composer wizard. Import this package into munki using munkiimport and then drop aforementioned command into post-install script.


For extra bonus points it would be cool to see if the root cert is installed before we go pushing this package to all the workstations in the network. To do this I added a little check install script to munki. Note that you have change the -c flag to match whatever you wrote for the root cert common name way back in step one.


security find-certificate -c "Root Cert Common Name" /Library/Keychains/System.keychain

if [ $? != 0 ]; then
exit 0
exit 1

You’ll note that the exit codes are reversed here, it’s because Munki will only install if the check install script exits on 0 which is how our security check command will exit if it finds the cert installed. So we flip the exit code to make munki do our bidding.

At this point you should be feeling like a rock star for the following reasons:

  • You haven’t given Go Daddy any money
  • You’ve successfully built and deployed your own cert authority
  • You won’t have to worry about onboarding new machines with the cert cause you’ve got in munki

Migrate DNS from Snow Leopard Server to OS X Server 5.x

Posted: February 8th, 2016 | Author: | Filed under: DNS, Mac OS X, Mac OS X Server | No Comments »

dns_bind-190x190I was faced with a DNS migration, but from Snow Leopard Server to 5.x. There were only 9 zones but there are hundreds of records and Apple provides zero tools to help make this migration easy. But, I found a hack. Now I’m going to say right now that I just found that this worked and YMMV.

First, on your Snow Leopard Server box do an ls /var/named/zones and make a primary zone in 5.x for every file listed in this directory. Then tarball up all these files and copy them over to your 5.x machine. One by copy the zones files from this tarball into /Library/Server/named matching the names as you go with some tab-auto-complete action.

For example, if the zone files in your Snow Leopard server are:

Then you would issue the following commands:

sudo cp /Library/Server/named/
sudo cp /Library/Server/named/
sudo cp /Library/Server/named/
sudo cp /Library/Server/named/
sudo cp /Library/Server/named/

Recovering Apple’s Wiki After Storage Failure or How I Learned to Love pg_resetxlog

Posted: February 2nd, 2016 | Author: | Filed under: Mac OS X, Mac OS X Server, Mountain Lion, postgres, Wiki | No Comments »

facepalmRecently I received a panicked phone call from a fellow sysadmin who was in a real jam. He had a customer who was dumping all their knowledge into Apple’s Wiki system running on top of Mountain Lion and Server 2.2.5. The storage system in the mini failed and they had to recover from backup, however the backup was setup using Carbon Copy Cloner and as we all know you cannot rely on a file-based backup system to backup a running postgres database.

After the data was restored the machine did boot but all the postgres services would not start, including the wiki. After reviewing the logs for quite some time I found some entries of pgstat wait timeout and then no log entries for about a day. I assumed that this was our hard drive failure window. Then two days later the log started producing tons of postgres crash statements, launchctl statements and this little nugget Jan 19th 13:29 database system was interrupted This was all the information I needed. From what I can tell, between the time that Carbon Copy Cloner calculated changes and the time that it copied the data some minute things changed within the database and so CCC didn’t get a proper clone. It appears that this error is caused when the database engine no longer knows where to start writing data back into the database. Basically, the counters were broken and had to be reset. Luckily postgres makes a tool called pg_resetxlog

The command has this basic structure:

-x XID set next transaction ID
-m XID set next multitransaction ID
-o OID set next OID
-l TLI,FILE,SEG force minimum WAL starting location for new transaction log

Now the Apple Wiki postgres data is held within /Library/Server/PostgreSQL\ For\ Server\ Services/Data which is an important detail to hold onto. Within this directory are all the bits of info you’ll need to run the following calculations. You’ll also need this decimal to hex converter.


A safe value for the next transaction ID (-x) can be determined by looking for the numerically largest file name in the directory pg_clog under the aforementioned postgres data directory, adding one, and then multiplying by 1048576. Note that the file names are in hexadecimal. It is usually easiest to specify the switch value in hexadecimal too. For example, if 0011 is the largest entry in pg_clog, -x 0x1200000 will work (five trailing zeroes provide the proper multiplier).

A safe value for the next multitransaction ID (-m) can be determined by looking for the numerically largest file name in the directory pg_multixact/offsets under the data directory, adding one, and then multiplying by 65536. As above, the file names are in hexadecimal, so the easiest way to do this is to specify the switch value in hexadecimal and add four zeroes.

A safe value for the next multitransaction offset (-O) can be determined by looking for the numerically largest file name in the directory pg_multixact/members under the data directory, adding one, and then multiplying by 65536. As above, the file names are in hexadecimal, so the easiest way to do this is to specify the switch value in hexadecimal and add four zeroes.

The WAL starting address (-l) should be larger than any WAL segment file name currently existing in the directory pg_xlog under the data directory. These names are also in hexadecimal and have three parts. The first part is the “timeline ID” and should usually be kept the same. Do not choose a value larger than 255 (0xFF) for the third part; instead increment the second part and reset the third part to 0. For example, if 00000001000000320000004A is the largest entry in pg_xlog, -l 0x1,0x32,0x4B will work; but if the largest entry is 000000010000003A000000FF, choose -l 0x1,0x3B,0x0 or more.

Once you have these four values you’re ready to try it out on your database. But before I began I requested a full bootable clone of the server as it was when they restored it, then I took this cloned and placed it into a VM in Fusion and snapped the VM before trying anything. Also, don’t forget that when you want to issue commands to the Apple postgres service you have to use the full path to the commands as well as use the _postgres user. My final command, which recovered the wiki system AND profile manager, looked like this:

sudo -u _postgres /Applications/ -f -x 0x100000 -m 0x10000 -o 0x10000 -l 0x1,0x2,0x18 /Library/Server/PostgreSQL\ For\ Server\ Services/Data

Feel free to reach out if you are having issues.

Using Munki’s nopkg to Push User Level Profiles

Posted: July 24th, 2015 | Author: | Filed under: Mac OS X, munki | Tags: | No Comments »

I recently needed to push some user level profiles, CardDAV to be specific. I use Meraki MDM but the custom mobileconfig profile would only install as a device profile. So I turned to my new munki install instead. Check out this post here if you’re not familiar with nopkg


First make sure you know the unique identifier of your profile, for an example we’ll use

First create a folder on your munki repo called profiles then copy the profile into this folder.

Use the following bash scripts in your pkginfo files to check, install, and uninstall the profile as you need

USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
sudo /usr/bin/profiles -P | grep | grep $USER
if [ $? -eq 0 ]; then
exit 1
exit 0

USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
/usr/bin/curl -L1 -o /tmp/profile.mobileconfig
sudo -u $USER /usr/bin/profiles -L -I -F /tmp/profile.mobileconfig
exit 0

USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
/usr/bin/curl -L1 -o /tmp/profile.mobileconfig
sudo -u $USER /usr/bin/profiles -L -R -F /tmp/profile.mobileconfig
exit 0

How to Automate FileMaker Server Fail Over on the Mac

Posted: September 24th, 2014 | Author: | Filed under: Filemaker, Mac OS X, Mac OS X Server | No Comments »

wool-clones-small-94006I have this managed services client, amazing client, easily my best one and my most favourite. Their workflow relies heavily upon their client roster database which is built on top of FileMaker. Recently I was doing their quarterly audit and noting all the single points of failure in the network. What I realized during this process was FileMaker Server running on top of a Mac Mini Server is a pretty big single point of failure. Of course I have FMServer doing regular backups but when confronted with the question: What is my recourse when the host running FM Server dies? The answer was, quickly install FM Server on a different machine, pull last night’s backup out of nearline storage, and put the server back on it’s feet. Sounds not too bad right? Wrong! There’s still another single point of failure.

Me. I’m busy man! I don’t have time to deal with fires every morning, if that’s how I worked I wouldn’t gone mental or quit years ago. I need automated server fail-over and without the ability to virtualize, due to budget not hardware, I was at a loss. Hmmmm what to do…. Maybe I should bash it? Maybe I should bash it so hard that at the end there’s bash script to bash it for me.

After a lot of bashing I have this script, and with the help of cron, every 15 mins it does a TCP connection to port 5003. It will try three times to connect to the port, if the port responds just once the script will then begin an rsync job of the remote database backup folder to local database backup folder and finally it will copy the latest backup into the databases folder. The idea here is to have the latest backup that was created on the primary FileMaker server to be the production database on the backup. Now if the script cannot connect to port 5003 on the primary then it fires the local FileMaker up and sends out an email alert and will continue to do so.

This script requires a few things:

Get the script off of github:

Note: there are a bunch of variables you’ll need to change at the top of the script. Once deployed ensure that the final result is a FMP12 file inside your replica database folder. If not edit the script, at the bottom you’ll find the rsync command, just tweak it for your environment. Post questions to comments.

Open Directory Replication 10.8.5 problems with Kerio Connnect 8.3.0

Posted: June 22nd, 2014 | Author: | Filed under: Kerberos, Kerio, LDAP, Mac OS X, Mac OS X Server, Mountain Lion, Open Directory | Tags: , , , , | No Comments »

kms_bubbleI recently was hired to implement an Open Directory Master/Replica into a network that wanted to leverage Kerio Connect mail server. At first, all seemed fine. I created the directory, the replica, and installed the kerio extension on both servers as was instructed by the fine folks at Kerio. Now I’d just like to say that this is different than what I remember in the days of 10.6. Back then you only had to install the OD extension on the master, the replica would then copy the schema over so that it could import the extended schema data at that time.

The problem comes into play when you have a master with already provisioned users in Kerio and you want to add an OD replica. Since the replica does not copy over the extended LDAP schema it is unable to replicate any provisioned users. The result is that those users will not exist in the replica which is bad news if you have services relying on that replica. To resolve this problem use the following procedure on the replica you wish to build:

sudo slapconfig -createreplica <master IP> diradmin

Once complete install the Kerio extention.

slapconfig -stopldapserver
slapadd -v -F /etc/openldap/slapd.d -c -l /var/db/openldap/openldap-data/backup.ldif
slapconfig -startldapserver


The New Customer Challenge

Posted: April 27th, 2014 | Author: | Filed under: Insight, Mac OS X, Mac OS X Server, Work | No Comments »

ticking-time-bombI’m an Apple consultant. I help small businesses who want nothing to do with the decision making aspect of technology. Planning, budgeting, procurement, deployment, support, deprecation, and recycling. Out of all these contexts no task is more challenging than workstations.

For those who are in the field, you know what I’m talking about. You get a new customer, they have workstations… some are new, some are old, some have MacKeeper, the bastard ones are carrying old migrated home folders that originated from 10.4 and a Cisco VPN kext. Some have 16 mail accounts filling 70% of the disk but since they’re “disabled” in you don’t see them at first. Now you have to dig to find out where the space is. Do this across 10 – 50 workstations and you will soon realize why I went bald early.

I needed a quick dirty way to get some very specific data out of the machine and into a little text file, yes I’m sure there are some sort of MDM tools or whatever might have you that will track everything that I don’t care about widget, but I don’t want that. It’s about workflow, see if I don’t get an idea of what I’m stepping into before I step into it I may find out something nasty far too late. In other words, I wouldn’t deploy an MDM before getting an idea of what’s going on.

Introducing feel free to download here

usage: -c <client name> -s <ftp server> -u <username> [-p <password>]
-c unique identifier for audit, a folder of this name will be made on your ftp server
-s ftp server fqdn/path sans protocol ie:
-u username to connect to ftp server
-p password for username, will prompt if none given
Requires root privileges to successfully deduce all features

Once I begin relations with the new customer I immediately gain admin access to all their machines, after placing the following script somewhere on the web I can then push it out through ARD in a script something like this:

curl -o /tmp/; chmod +x /tmp/; /tmp/ -c clientname -s -u ftpuser -p "ftpuserpass"

I also have it wrapped in AppleScript so that I can pop it over email to any remote machines. Usually also along with a Meraki MDM as well. Just place this code into Script Editor, then save as an application. Place inside the package of the AppleScript app.

## change the switch arguments!
set path_ to (path to me as string)
set p to POSIX path of path_
do shell script "" & p & "/ -c clientname -s -u ftpuser -p 'ftpuserpass'" with administrator privileges

Automated Backups of Mac OS X Server 2.2.2

Posted: April 27th, 2014 | Author: | Filed under: DNS, Mac OS X, Mac OS X Server, Mountain Lion, Open Directory | No Comments »

Hi Everybody! dr-nick-riviera

So I’ve been in the Mac game for quite some time now and all along I was always longing for a good automated backup solution. A few years ago myself and a colleague got together and wrote A simple shell script with a few variables inside. Simply edit the shell script and then install as a cronjob to run nightly. Features of this backup script include:

  • Open Directory archiving
  • Service Plists
  • CalDAV/CardDAV database
  • Profile Manager database
  • DNS records
  • Wiki database and binary files
  • Webmail

I’ve been using this script for years now under 10.6, 10.7 and 10.8. The version listed here is for Server 2.2.2 under 10.8.5

Restoration of these backups is fairly simple to do as long as you know some postgres commands. Here’s the article on how to restore the wiki.

Calendar, webmail are fairly similar. DNS restoration is just a matter of placing the files back in /var/named and /etc/named.conf

If you need to restore open directory archive you should use Apple’s latest knowledge base instructions. Just make sure that the server hostname matches the backup.

To restore OS X Server setting plists:

sudo serveradmin settings < /path/to/your-sa_backup-servicename-plist

Get the code here.

How to Rebuild Software RAID 10 in Mountain Lion – Command Line

Posted: February 27th, 2014 | Author: | Filed under: Mac OS X, Mac OS X Server, RAID | No Comments »


First, replace the disk 😉

Open a terminal session on affected system and run
diskutil appleRAID list
find the UUID for the affected drive and the UUID for theRAID set. Then fill in the blanks below:
sudo diskutil appleRAID remove <drive UUID> <RAID UUID>
For example:
sudo diskutil appleRAID remove EEDD0AD6-C448-48F7-A766-001C65338E99 7010C337-829C-4F08-B6A4-1C8A9E943CBD
Our RAID 10 now only has three disks attached.


First we need to identify the spare disk waiting for us in the system. Use diskutil list to do this. Here’s some example output. See if you can spot the disk that is not like the others.

# diskutil list

   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *119.9 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS ServerHD                118.9 GB   disk0s2
   3:                 Apple_Boot Recovery HD             784.2 MB   disk0s3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                 Apple_RAID                         2.0 TB     disk1s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk1s3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                 Apple_RAID                         2.0 TB     disk2s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk2s3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk3
   1:                        EFI                         209.7 MB   disk3s1
   2:                 Apple_RAID                         2.0 TB     disk3s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk3s3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *2.0 TB     disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Storage                *4.0 TB     disk5
thuja:~ pwladmin$


We can see in this list that disk4 has no partition maps and thus is the new disk. We can now add this disk into our degraded raid with:
sudo diskutil appleRAID add member <NewMemberDeviceName> <RAID UUID>
for example
sudo diskutil appleRAID add member disk4 7010C337-829C-4F08-B6A4-1C8A9E943CBD
Your disk is now part of the raid set. If you use diskutil appleRAID list you’ll be able to check the progress of the rebuild.

Name: Thuja RAID A
Unique ID: 920F03EB-DE44-49AA-9934-0EF53EF032D1
Type: Mirror
Status: Online
Size: 2.0 TB (2000054910976 Bytes)
Rebuild: manual
Device Node: -
# DevNode UUID Status Size
0 disk1s2 D4BCB349-3255-473B-B586-EAF066C5BD6D Online 2000054910976
1 disk3s2 E01DB36B-CDC4-458C-AC07-507433DCB481 Online 2000054910976
Name: Thuja Stripe RAID
Unique ID: 9D9FEE5F-5F04-4051-A0AB-A985DFFAF2A0
Type: Stripe
Status: Online
Size: 4.0 TB (4000109559808 Bytes)
Rebuild: manual
Device Node: disk5
# DevNode UUID Status Size
0 -none- 920F03EB-DE44-49AA-9934-0EF53EF032D1 Online 2000054779904
1 -none- 7010C337-829C-4F08-B6A4-1C8A9E943CBD Online 2000054779904
Name: Thuja RAID B
Unique ID: 7010C337-829C-4F08-B6A4-1C8A9E943CBD
Type: Mirror
Status: Degraded
Size: 2.0 TB (2000054910976 Bytes)
Rebuild: manual
Device Node: -
# DevNode UUID Status Size
0 disk2s2 13AFF0CD-77FB-4E14-9A89-A09C01ACA4C4 Online 2000054910976
1 disk4s2 EAE79161-3729-41FB-81A1-97CE878C1E31 1% (Rebuilding)2000054910976


CrashPlan Proe 3.8.2010.2 on Mountain Lion 10.8.5

Posted: February 22nd, 2014 | Author: | Filed under: CrashPlan, Mac OS X | 1 Comment »

I recently had an issue where I could not load the CrashPlan proe server onto a 10.8.5 Mac Mini. The app installed however when I asked for the management interface on port 4280 I was greeted with a URL redirect and a blank white page. I was also getting this from /Library/Logs/PROserver/proserver.startup.err

[02.22.14 11:45:02.159 INFO main temPropertiesLoader.loadSystemProperties] * loading properties from: conf/
com.code42.exception.DebugRuntimeException: Failed to start CPCentralServices.
at com.backup42.controller.CPCentralController.start(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at com.backup42.main.CPServiceManager.invokeAll(
at com.backup42.main.CPServiceManager.start(
at com.backup42.main.CPServer.start(
at com.backup42.main.CPServer.main(
Caused by: java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(
at java.lang.Long.parseLong(
at java.lang.Long.valueOf(
at com.backup42.server.manage.ServerManager.initializeMyGuid(
at com.backup42.server.manage.OsXServerManager.initializeGuid(
at com.backup42.server.manage.ServerManagerService.initializeGuid(

The solution was the following:

stop the service launchctl unload /Library/LaunchDaemons/com.crashplan.proserver.plist
edit launchd plist to point to /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Commands/java instead of /usr/bin/java
then rm /Library/CrashPlan/.proserver_identity
start the service and profit!