Migrate DNS Service from old host to new host – Server 5.x & 6.x

Posted: September 10th, 2017 | Author: | Filed under: DNS, Mac OS X, macOS Sierra | No Comments »

1F2F983500000578-3499652-image-a-8_1458344126936It’s so easy.

On your source host simply copy the contents of /Library/Server/named to same directory on the new host. Refresh server.app and don’t forget to change the NS records to point to the new host.

Bam


Why you should *never* leave your Mac unattended

Posted: July 13th, 2017 | Author: | Filed under: Mac OS X | No Comments »

insecureRecently I was contracted by a local backpack company to automate changing some settings on a bunch of Mac clients. The three main parts to the script had to; clean the connect to servers window and inject a new server record in to favorites, delete all login items referencing an old file server, and find a specific keychain item, retrieve the password, and inject a new keychain item using the same password. The first two were easy enough but the keychain was interesting. Through a little bit of research I soon found that the security command could do most of what I wanted. If you issued a command like

security find-internet-password -a jordan -s file.server.local -w

It would prompt you to either allow or deny access to the item and then spit the password out on the command line. But wait a minute here, in Keychain Access.app if I want to show a password I would need the password to the computer to gain access. Not on the command line though. This was pretty interesting I thought. That means as long as I have access to the console and the machine is logged in I can extract *any* password from the login.keychain that I want.

To take this a step further I made a little bash script that dumps the login.keychain to file using security dump-keychain login.keychain then I parse the data finding all the entries that I can retrieve the password for. The script will prompt the user once for each item that it finds in the keychain and then write all the data out to a csv file in /tmp

Is this a security flaw? Well, the jury’s out on that one. Some people say YES! and some people say no. To me, it’s all just rather interesting. Try it out for yourself and you’ll see why you should never leave your machine unattended again.

Download script here

Oh btw, I am *not* a developer. Amateur scripter at best, this code takes a long time to run. Upwards 10 mins for larger keychains.


Mac OS X Server 5.x El Capitan SSL Certificate Authority

Posted: February 11th, 2016 | Author: | Filed under: Mac OS X, Mac OS X Server, SSL | 3 Comments »

280px-El_Capitan_2005I use Open Directory a lot, I can’t think of many times when I don’t use it at least once a day in some way whether that be direct or indirect. It’s not the best directory system out there – hell, it’s not even very good but it’s what the fine people at Apple have supplied us with and it’s what I use. Although I have long had a pet peeve with the way OD is built if you just run through the setup wizard. The certificate expires in one year. Almost every time I encounter an OD server in the wild nine times out of ten the certs are a mess. They’re either expired or about to expire and all the services that depend on them are freaking out. The clients are constantly being prompted to accept an invalid cert and OD fail-over tends to stop working. My solution has been to build a certificate authority for all my OD installs and to build my own certs that are valid for ten years, that way I won’t have to worry about them expiring and let’s be honest there’s no Mac server on the planet that’s going to last ten years lol (sense the cynicism yet?)

Create the Root Key

First we’re going to hop into a terminal on any Mac OS X box and navigate to somewhere safe in the file system and build the master key, make sure the password you use for this is kept secret and safe.

cd /some/where/safe/I/dont/care/where
mkdir -p myCA/cert myCA/key
cd myCA
openssl genrsa -aes256 -out key/ca.key.pem 4096
chmod 400 key/ca.key.pem

That OpenSSL command is going to ask you a bunch of questions, answer them to the best way you see fit and for the common name put the organization that you’re building this certificate authority for. Do not put an actual domain. Record what you wrote for Common Name because we’re going to need it later. Whitespaces are allowed. Oh and one more pro tip, make sure the value of the certificate common is unique and can’t be half-matched. For example I made a root cert called “Client Name” but when it came to do deploy the certificate via munki, the check install script found a cert called “Client Name Open Directory” and thus the command matched and wouldn’t deploy the new root cert.

Create the Root Cert


openssl req -key key/ca.key.pem -new -x509 -days 3650 -sha256 -extensions v3_ca -out cert/cert.pem
chmod 444 cert/cert.pem

Now we have two files.

cert/cert.pem – This is your CA’s certificate and can be publicly available and of course world readable. You will need to load this certificate into all the clients in your network.

key/ca.key.pem – This is your CA’s private key. Although it is protected with a passphrase you should restrict access to it, so that only root can read it.

Create the First Server key/cert combo

We can now start creating SSL certificates for our various servers and services. Create a directory named a hostname corresponding to the hostname for the computer or service you are creating the certificate for.

Create the first key

openssl genrsa -aes256 -out hostname/hostname.domainname.key 4096

While answering the questions for this make sure you type in the FQDN for server or service you want to secure into Common Name.

Create the server cert

openssl req -new -key hostname/hostname.domainname.key -out hostname/hostname.domainname.csr

Sign the cert

openssl x509 -req -days 3650 -in hostname/hostname.domainname.csr -CA cert/cert.pem -CAkey key/ca.key.pem -set_serial 01 -out hostname/hostname.domainname.crt

Now before we continue I have to drill into you that it’s imperative that you document these certs as you make them, which hosts they’re deployed on to, and what the serial number is. It will help you in the long run.

Finally we’re going to make a passwordless version of the server key, this is the key that we’ll ultimately use on our server. We need a passwordless key so that the Mac OS X services do not need human intervention when trying to use the certs. Otherwise you’ll have to type the cert password in every time you restart the service.


openssl rsa -in hostname/hostname.domainname.key -out hostname/hostname.domainname.key.insecure
mv hostname/hostname.domainname.key hostname/hostname.domainname.key.secure
mv hostname/hostname.domainname.key.insecure hostname/hostname.domainname.key

Import the CA and Cert

Copy the myCA/cert.pem, hostname/hostname.domainname.key, and hostname/hostname.domainname.crt files to your Mac OS X server. Double click on the myCA/cert.pem, this should open the file with Keychain Access which will ask you which keychain to import the cert to. Select the system keychain and then double click on the entry in Keychain Access and set the trust setting to Always Trust.

Next open Server.app and click on Certificates. Click on the gear menu and select Import Certificate Identity. Drag in the hostname/hostname.domainname.key, and hostname/hostname.domainname.crt files, when you do this you should see that the files are signed by the organization that you entered in step one. Finally select the cert from the list of certs in server.app and give it time to switch over.

Finally fire up the wiki service in server.app, whip open your browser of choice and connect to the FQDN that you create the SSL cert for. Remember, if your cert common name doesn’t match the way you address the wiki in the URL bar you will get a hostname mismatch error. You MUST make sure the names match from what you type into the certificate’s common name and how you connect to the wiki.

Root Cert Deployment

So now maybe you want to be able to use this cert with some of your clients right? You can do this a multitude of ways. Such as:

Copy the file to /tmp across your network and run the following command as root


security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "/tmp/cert.pem"; srm "/tmp/cert.pem"

Or if you’re cool you’ll use some sort of package deployment system. I use munki because it’s not backed by some money hungry corporation. Here’s looking at you Cohen. 😉 To do this I made a new package with Composer dropped my root cert into /tmp and then wrapped then finished the Composer wizard. Import this package into munki using munkiimport and then drop aforementioned command into post-install script.

 

For extra bonus points it would be cool to see if the root cert is installed before we go pushing this package to all the workstations in the network. To do this I added a little check install script to munki. Note that you have change the -c flag to match whatever you wrote for the root cert common name way back in step one.


#!/bin/bash

security find-certificate -c "Root Cert Common Name" /Library/Keychains/System.keychain

if [ $? != 0 ]; then
exit 0
else
exit 1
fi

You’ll note that the exit codes are reversed here, it’s because Munki will only install if the check install script exits on 0 which is how our security check command will exit if it finds the cert installed. So we flip the exit code to make munki do our bidding.

At this point you should be feeling like a rock star for the following reasons:

  • You haven’t given Go Daddy any money
  • You’ve successfully built and deployed your own cert authority
  • You won’t have to worry about onboarding new machines with the cert cause you’ve got in munki


Migrate DNS from Snow Leopard Server to OS X Server 5.x

Posted: February 8th, 2016 | Author: | Filed under: DNS, Mac OS X, Mac OS X Server | No Comments »

dns_bind-190x190I was faced with a DNS migration, but from Snow Leopard Server to Server.app 5.x. There were only 9 zones but there are hundreds of records and Apple provides zero tools to help make this migration easy. But, I found a hack. Now I’m going to say right now that I just found that this worked and YMMV.

First, on your Snow Leopard Server box do an ls /var/named/zones and make a primary zone in Server.app 5.x for every file listed in this directory. Then tarball up all these files and copy them over to your Server.app 5.x machine. One by copy the zones files from this tarball into /Library/Server/named matching the names as you go with some tab-auto-complete action.

For example, if the zone files in your Snow Leopard server are:


db.1.5.10.in-addr.arpa.zone.apple
db.mgmt.clientname.com.zone.apple
db.remote.clientname.com.zone.apple
db.backup.clientname.com.zone.apple
db.lan.clientname.com.zone.apple

Then you would issue the following commands:


sudo cp db.1.5.10.in-addr.arpa.zone.apple /Library/Server/named/db.1.5.10.in-addr.arpa
sudo cp db.lan.clientname.com.zone.apple /Library/Server/named/db.lan.clientname.com
sudo cp db.mgmt.clientname.com.zone.apple /Library/Server/named/db.mgmt.clientname.com
sudo cp db.remote.clientname.com.zone.apple /Library/Server/named/db.remote.clientname.com
sudo cp db.backup.clientname.com.zone.apple /Library/Server/named/db.backup.clientname.com


Recovering Apple’s Wiki After Storage Failure or How I Learned to Love pg_resetxlog

Posted: February 2nd, 2016 | Author: | Filed under: Mac OS X, Mac OS X Server, Mountain Lion, postgres, Wiki | No Comments »

facepalmRecently I received a panicked phone call from a fellow sysadmin who was in a real jam. He had a customer who was dumping all their knowledge into Apple’s Wiki system running on top of Mountain Lion and Server 2.2.5. The storage system in the mini failed and they had to recover from backup, however the backup was setup using Carbon Copy Cloner and as we all know you cannot rely on a file-based backup system to backup a running postgres database.

After the data was restored the machine did boot but all the postgres services would not start, including the wiki. After reviewing the logs for quite some time I found some entries of pgstat wait timeout and then no log entries for about a day. I assumed that this was our hard drive failure window. Then two days later the log started producing tons of postgres crash statements, launchctl statements and this little nugget Jan 19th 13:29 database system was interrupted This was all the information I needed. From what I can tell, between the time that Carbon Copy Cloner calculated changes and the time that it copied the data some minute things changed within the database and so CCC didn’t get a proper clone. It appears that this error is caused when the database engine no longer knows where to start writing data back into the database. Basically, the counters were broken and had to be reset. Luckily postgres makes a tool called pg_resetxlog

The command has this basic structure:

pg_resetxlog
-x XID set next transaction ID
-m XID set next multitransaction ID
-o OID set next OID
-l TLI,FILE,SEG force minimum WAL starting location for new transaction log
/path/to/database/directory

Now the Apple Wiki postgres data is held within /Library/Server/PostgreSQL\ For\ Server\ Services/Data which is an important detail to hold onto. Within this directory are all the bits of info you’ll need to run the following calculations. You’ll also need this decimal to hex converter.

Source: http://www.postgresql.org/docs/9.0/static/app-pgresetxlog.html

A safe value for the next transaction ID (-x) can be determined by looking for the numerically largest file name in the directory pg_clog under the aforementioned postgres data directory, adding one, and then multiplying by 1048576. Note that the file names are in hexadecimal. It is usually easiest to specify the switch value in hexadecimal too. For example, if 0011 is the largest entry in pg_clog, -x 0x1200000 will work (five trailing zeroes provide the proper multiplier).

A safe value for the next multitransaction ID (-m) can be determined by looking for the numerically largest file name in the directory pg_multixact/offsets under the data directory, adding one, and then multiplying by 65536. As above, the file names are in hexadecimal, so the easiest way to do this is to specify the switch value in hexadecimal and add four zeroes.

A safe value for the next multitransaction offset (-O) can be determined by looking for the numerically largest file name in the directory pg_multixact/members under the data directory, adding one, and then multiplying by 65536. As above, the file names are in hexadecimal, so the easiest way to do this is to specify the switch value in hexadecimal and add four zeroes.

The WAL starting address (-l) should be larger than any WAL segment file name currently existing in the directory pg_xlog under the data directory. These names are also in hexadecimal and have three parts. The first part is the “timeline ID” and should usually be kept the same. Do not choose a value larger than 255 (0xFF) for the third part; instead increment the second part and reset the third part to 0. For example, if 00000001000000320000004A is the largest entry in pg_xlog, -l 0x1,0x32,0x4B will work; but if the largest entry is 000000010000003A000000FF, choose -l 0x1,0x3B,0x0 or more.

Once you have these four values you’re ready to try it out on your database. But before I began I requested a full bootable clone of the server as it was when they restored it, then I took this cloned and placed it into a VM in Fusion and snapped the VM before trying anything. Also, don’t forget that when you want to issue commands to the Apple postgres service you have to use the full path to the commands as well as use the _postgres user. My final command, which recovered the wiki system AND profile manager, looked like this:

sudo -u _postgres /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_resetxlog -f -x 0x100000 -m 0x10000 -o 0x10000 -l 0x1,0x2,0x18 /Library/Server/PostgreSQL\ For\ Server\ Services/Data

Feel free to reach out if you are having issues.


Using Munki’s nopkg to Push User Level Profiles

Posted: July 24th, 2015 | Author: | Filed under: Mac OS X, munki | Tags: | No Comments »

I recently needed to push some user level profiles, CardDAV to be specific. I use Meraki MDM but the custom mobileconfig profile would only install as a device profile. So I turned to my new munki install instead. Check out this post here if you’re not familiar with nopkg http://grahamgilbert.com/blog/2014/07/27/personal-automation-munki-part-2/

 

First make sure you know the unique identifier of your profile, for an example we’ll use com.company.carddav

First create a folder on your munki repo called profiles then copy the profile into this folder.

Use the following bash scripts in your pkginfo files to check, install, and uninstall the profile as you need

<key>installcheck_script</key>
<string>#!/bin/bash
USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
sudo /usr/bin/profiles -P | grep com.company.carddav | grep $USER
if [ $? -eq 0 ]; then
exit 1
else
exit 0
fi


<key>postinstall_script</key>
<string>#!/bin/bash
USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
/usr/bin/curl -L1 http://munki.yourmunkirepo.com/profiles/com.company.carddav.mobileconfig -o /tmp/profile.mobileconfig
sudo -u $USER /usr/bin/profiles -L -I -F /tmp/profile.mobileconfig
exit 0


<key>uninstall_method</key>
<string>uninstall_script</string>
<key>uninstall_script</key>
<string>#!/bin/bash
USER=`/usr/bin/who | grep console | cut -d ' ' -f1`
/usr/bin/curl -L1 http://munki.yourmunkirepo.com/profiles/com.company.carddav.mobileconfig -o /tmp/profile.mobileconfig
sudo -u $USER /usr/bin/profiles -L -R -F /tmp/profile.mobileconfig
exit 0


How to Automate FileMaker Server Fail Over on the Mac

Posted: September 24th, 2014 | Author: | Filed under: Filemaker, Mac OS X, Mac OS X Server | No Comments »

wool-clones-small-94006I have this managed services client, amazing client, easily my best one and my most favourite. Their workflow relies heavily upon their client roster database which is built on top of FileMaker. Recently I was doing their quarterly audit and noting all the single points of failure in the network. What I realized during this process was FileMaker Server running on top of a Mac Mini Server is a pretty big single point of failure. Of course I have FMServer doing regular backups but when confronted with the question: What is my recourse when the host running FM Server dies? The answer was, quickly install FM Server on a different machine, pull last night’s backup out of nearline storage, and put the server back on it’s feet. Sounds not too bad right? Wrong! There’s still another single point of failure.

Me. I’m busy man! I don’t have time to deal with fires every morning, if that’s how I worked I wouldn’t gone mental or quit years ago. I need automated server fail-over and without the ability to virtualize, due to budget not hardware, I was at a loss. Hmmmm what to do…. Maybe I should bash it? Maybe I should bash it so hard that at the end there’s bash script to bash it for me.

After a lot of bashing I have this script, and with the help of cron, every 15 mins it does a TCP connection to port 5003. It will try three times to connect to the port, if the port responds just once the script will then begin an rsync job of the remote database backup folder to local database backup folder and finally it will copy the latest backup into the databases folder. The idea here is to have the latest backup that was created on the primary FileMaker server to be the production database on the backup. Now if the script cannot connect to port 5003 on the primary then it fires the local FileMaker up and sends out an email alert and will continue to do so.

This script requires a few things:

Get the script off of github: https://github.com/syntaxcollector/syncFM

Note: there are a bunch of variables you’ll need to change at the top of the script. Once deployed ensure that the final result is a FMP12 file inside your replica database folder. If not edit the script, at the bottom you’ll find the rsync command, just tweak it for your environment. Post questions to comments.


Open Directory Replication 10.8.5 problems with Kerio Connnect 8.3.0

Posted: June 22nd, 2014 | Author: | Filed under: Kerberos, Kerio, LDAP, Mac OS X, Mac OS X Server, Mountain Lion, Open Directory | Tags: , , , , | No Comments »

kms_bubbleI recently was hired to implement an Open Directory Master/Replica into a network that wanted to leverage Kerio Connect mail server. At first, all seemed fine. I created the directory, the replica, and installed the kerio extension on both servers as was instructed by the fine folks at Kerio. Now I’d just like to say that this is different than what I remember in the days of 10.6. Back then you only had to install the OD extension on the master, the replica would then copy the schema over so that it could import the extended schema data at that time.

The problem comes into play when you have a master with already provisioned users in Kerio and you want to add an OD replica. Since the replica does not copy over the extended LDAP schema it is unable to replicate any provisioned users. The result is that those users will not exist in the replica which is bad news if you have services relying on that replica. To resolve this problem use the following procedure on the replica you wish to build:

sudo slapconfig -createreplica <master IP> diradmin

Once complete install the Kerio extention.

slapconfig -stopldapserver
slapadd -v -F /etc/openldap/slapd.d -c -l /var/db/openldap/openldap-data/backup.ldif
slapconfig -startldapserver

#gowellandinpiece
#replication


The New Customer Challenge

Posted: April 27th, 2014 | Author: | Filed under: Insight, Mac OS X, Mac OS X Server, Work | No Comments »

ticking-time-bombI’m an Apple consultant. I help small businesses who want nothing to do with the decision making aspect of technology. Planning, budgeting, procurement, deployment, support, deprecation, and recycling. Out of all these contexts no task is more challenging than workstations.

For those who are in the field, you know what I’m talking about. You get a new customer, they have workstations… some are new, some are old, some have MacKeeper, the bastard ones are carrying old migrated home folders that originated from 10.4 and a Cisco VPN kext. Some have 16 mail accounts filling 70% of the disk but since they’re “disabled” in Mail.app you don’t see them at first. Now you have to dig to find out where the space is. Do this across 10 – 50 workstations and you will soon realize why I went bald early.

I needed a quick dirty way to get some very specific data out of the machine and into a little text file, yes I’m sure there are some sort of MDM tools or whatever might have you that will track everything that I don’t care about widget, but I don’t want that. It’s about workflow, see if I don’t get an idea of what I’m stepping into before I step into it I may find out something nasty far too late. In other words, I wouldn’t deploy an MDM before getting an idea of what’s going on.

Introducing sysAudit.sh: feel free to download here

usage: sysAudit.sh -c <client name> -s <ftp server> -u <username> [-p <password>]
OPTIONS:
-c unique identifier for audit, a folder of this name will be made on your ftp server
-s ftp server fqdn/path sans protocol ie: mybigfat.ftpserver.com
-u username to connect to ftp server
-p password for username, will prompt if none given
NOTE:
Requires root privileges to successfully deduce all features

Once I begin relations with the new customer I immediately gain admin access to all their machines, after placing the following script somewhere on the web I can then push it out through ARD in a script something like this:

curl -o /tmp/sysAudit.sh http://www.copiousit.com/sysAudit.sh; chmod +x /tmp/sysAudit.sh; /tmp/sysAudit.sh -c clientname -s ftp.server.com -u ftpuser -p "ftpuserpass"

I also have it wrapped in AppleScript so that I can pop it over email to any remote machines. Usually also along with a Meraki MDM as well. Just place this code into Script Editor, then save as an application. Place sysAudit.sh inside the package of the AppleScript app.

## change the switch arguments!
set path_ to (path to me as string)
set p to POSIX path of path_
do shell script "" & p & "/sysAudit.sh -c clientname -s ftp.server.com -u ftpuser -p 'ftpuserpass'" with administrator privileges


Automated Backups of Mac OS X Server 2.2.2

Posted: April 27th, 2014 | Author: | Filed under: DNS, Mac OS X, Mac OS X Server, Mountain Lion, Open Directory | No Comments »

Hi Everybody! dr-nick-riviera

So I’ve been in the Mac game for quite some time now and all along I was always longing for a good automated backup solution. A few years ago myself and a colleague got together and wrote osx-backup.sh. A simple shell script with a few variables inside. Simply edit the shell script and then install as a cronjob to run nightly. Features of this backup script include:

  • Open Directory archiving
  • Service Plists
  • CalDAV/CardDAV database
  • Profile Manager database
  • DNS records
  • Wiki database and binary files
  • Webmail

I’ve been using this script for years now under 10.6, 10.7 and 10.8. The version listed here is for Server 2.2.2 under 10.8.5

Restoration of these backups is fairly simple to do as long as you know some postgres commands. Here’s the article on how to restore the wiki.

Calendar, webmail are fairly similar. DNS restoration is just a matter of placing the files back in /var/named and /etc/named.conf

If you need to restore open directory archive you should use Apple’s latest knowledge base instructions. Just make sure that the server hostname matches the backup.

To restore OS X Server setting plists:

sudo serveradmin settings < /path/to/your-sa_backup-servicename-plist

Get the code here.