iDent, Or That Time I Accidentally Recreated Fastlane Match

Credit: CC0 Public Domain

So, I work at a big company. Because my team builds and deploys all the iOS apps for our entire line of business, we’ve gotten a hold of 40 separate build nodes we’ve hooked into our Jenkins. We literally have more build real estate than we know what to do with. The downside? When I arrived on the team, we were maintaining those servers completely by hand. No shared configuration management, no automated reloads, nothing.

Most pertinent to this piece, though, was that we were managing signing certificates and provisioning profiles manually. Anyone who’s worked on multiple macs to build and deliver iOS apps can feel my pain here. Our workflow had been to upload the files manually to a specific node, which was usually dedicated to a given app (and only that app). There was no unified rollout between all 40 nodes, no version control, and we kept only minimal, hand updated records as to what version was stored where. After one particularly grueling sprint, I put my foot down and decided: no more! I was going to roll a solution to this or die trying.

Luckily, I was able to pull off the former. Coming into the project, I knew I needed five things:

1. I needed to store the profiles and certificates (which for ease of use I will refer to collective as “identities”) in a version controlled, central location.
2. I wanted a self-service option for anyone to upload an identity into our repository.
3. I needed to propagate any changes to a given identity out across all 39 build nodes.
4. I needed to run regular checks to determine whether an identity was expired.
5. I wanted to have regular cleanups that would delete expired identities, and open up JIRA tickets as reminders to renew soon-to-expire identities.

It was only after I was just about done with my solution that I realized I had rebuilt Fastlane Match. The only problem with using that popular open-source tool was it didn’t seem designed to work across multiple teams. My team builds a half dozen iOS apps, across three separate Team IDs/Developer Accounts. Further, corporate policy demanded that my team doesn’t generate new identities, so the deletion and re-registering of new identities with match was right out as well. In the end, I just couldn’t figure out how to just set and forget Match (Felix Krause, if you’re reading this, I am obviously all ears).

With all this in mind, let’s start working through how we solved the five requirements above.

Version-Controlled Central Repo

This facet was by far the easiest. I set up a bitbucket repo with access limited to just my team. Really only minor fuss required. I ended up calling it iDent, cause I wanted a snappy name that could stick in my team’s mind. Jury’s obviously still out on whether I succeeded.

Self-Service option

Since my team manages a Jenkins instance, we wanted to have our first pass at this expose self-service options through it. We’re already known for the instance, so disseminating info on its existence wasn’t going to be too hard. I set up two jobs: one for uploading profiles, and one for uploading signing certificates.

Common Features

Largely, the upload mechanism for both jobs was identical: the Jenkins job accepted a file as a parameter, and kicked off a shell script on one of our “admin” boxes. The script would check if the uploaded file was appropriate for the job, and fail the build if it was not. After that, it would checkout a new branch for the iDent and run a secondary script that posted to bitbucket’s api, opening up a PR into the central repo. The script would automatically tag my teammates in the PR, and from there, we could choose whether to approve or deny the upload.

Unique Challenges for Signing Certs

Part and parcel of accepting the signing certs for our build nodes was wrapping it up in a p12. Now, by convention, they would be passed into the Jenkins job as a pkcs12 file which wrapped both the cert and its associated signing keys. However, we still had to accept the end-user’s arbitrary p12 password in order to be able to use it at all. We ended up solving this by simply laundering the p12 file: first, we’d create a new keychain on the admin box. Then, we’d import the p12 onto to it, before exporting it back out—replacing the previous password with our enterprise password. Thankfully, we automated all of this with the security cli, otherwise I would’ve gone completely insane.

Change Propagation

Fortunately for us, change propagation was one of the easier steps. We had another Jenkins job (not self-service, mind you) set up to poll against changes to master in the central iDent repo. On commits, we’d kick up a shell script on each of our 40 build nodes, pulling down the latest changes to iDent into a temp local folder. From there, we rsync’d over the provisioning profiles to ~/Library/MobileDevice/Provisioning\ Profiles and imported the signing certificates directly onto the enterprise keychain. The temp local folder would then be deleted. No muss, no fuss.

Expiration Checks

Now we come to the second of our non-self serve Jenkins jobs: the expiration checker. In implementation, points 4 & 5 ended up being easily combinable, but I will keep the sections separate for ease of explanation. The Expiration Checker kicks off a single shell script, cleanup-orchestrator. It runs through four loops: iterating once over every profile in ~/Library/MobileDevice/Provisioning\ Profiles, once over every cert on the enterprise keychain, and once on each profile and identity checked into the repo itself¹. Openssl x509 , thankfully, provides functionality to extract the expiration dates of both. From there, it was just a case of refining the date down to YYYY-MM-DD format, getting the current date, and comparing.

Handling Expiration

Next step, we had to handle that expiration. Still part of the same script, we would check if the identity was set to expire in 31 days. If so, we kicked off a reminder-to-renew script. In iDent’s initial state, this script simply opened a ticket in my team’s ServiceDesk queue. While not the perfect self-serve reminder that I initially envisioned, it served well enough to get the project off the ground. The case for when an identity was already expired is, obviously, much simpler.

You delete it.

Finishing Touches

Now that we covered my initial five goals, we were well on our way to finishing iDent. Much as is often the case, though, as we rolled out the service, we identified further pain points we wanted to solve. Namely, erasing expired identities from the central repo, and how to handle truly self-serve uploads.

Reaching Back

Reaching back to delete the expired identities from the repo was pretty straightforward. We already had a script to open up PRs in the iDent repo; we simply added a call to it after the expiration checks were done, but before the workspace was cleared. Otherwise, the workflow was identical to the initial uploads.

True Self-Service

The addition of true self-service was the first major version bump of the project, as well as the first big contribution to it from another developer. We knew we wanted to collect the email address of the uploading end-users. We knew we wanted to record that email address in direct correlation to the identity that was uploaded. Finally, we knew that record had to be checked into the iDent repo alongside everything else.

My co-workers solution was ingenious: 1) he tracked the correlation in a yaml manifest, 2) he wrote a python wrapper that allowed seamless updating of the manifest, and 3) he wrote another wrapper that drafted up the jinja templates to send out as emails to alert the end users. From there, it was simply a matter of slotting the new self-service functionality into my shell scripts, and rolling the changes out.

iDentical Outcomes

Almost immediately after implementation, we started to see dividends. What had previously been untracked ad-hoc uploads now had centralized records and automated reminders. No longer did we get our first warnings about identity expiration with failing Jenkins jobs. The whole effort took about a month to complete, but I’d like to think it pushed us forward years in proper DevOps maturation.

The best part, though, was with all our identities equalized, the door was open to a much more ambition modernization effort. Without identities to pin us to specific nodes, we soon started making our first steps into completely unpinning our build jobs. While, obviously, that should’ve been Step One of a modernized build environment, we’ve managed to unblock ourselves from a lot of headaches by rolling iDent. If only a month of work can catapult us from the Dark Ages to The Cloud, imagine what we can do with a little more time!

1. The duplicated loops are a necessary evil: while it would be all well and good to only delete the node’s copies of the identities (be they in ~/Library or on the keychain), if they just get imported again on the next upload, there’s no real point. They’ll just end up taking up space. For more on how we solved this, check out the Finishing Touches section. Go back