Under the Hood of mitcnc.org

Posted on November 28, 2020 · 15 mins read

Over the past six months I’ve been working on changes to mitcnc.org, the homepage of the MIT Club of Northern California (MITCNC), and the portal through which alumni join the club and register for events. During this time we’ve updated our SSO integration with the MIT Alumni Association, loaded new data to our data warehouse (hosted in BigQuery), changed server configurations, improved our deployment and observability processes, and moved our memberships processing to Stripe Billing. This has been quite a long journey with many ups and downs.

We have done all of this to improve the experience of our fellow alumni. It is worth noting that MITCNC is unique amongst the MIT alumni clubs and affinity groups: we are the only club that maintains its own website. All other clubs rely on iModules, run by the MIT Alumni Association (MITAA), for their website/CMS, event registration, membership management, and email marketing. The system is notoriously slow—think 5-9 second page loads—and the UI is unintuitive. Yet, there are no great off-the-shelf alternatives.

I began exploring alternatives years ago when I was president of the Club of Boston (MITCOB), and came up short. iModules appears to be the most popular software for alumni engagement despite its many flaws. If you break it apart, you could build a system that encompasses a good CMS and relies on existing solutions for the rest: Eventbrite for events, MailChimp for email marketing, and Stripe for subscriptions. However, ditching iModules means making a commitment to not only build a custom solution, but maintain it for years to come. It is hard to make this commitment when club leadership shifts every 1-2 years, and volunteers come and go. MITCNC is the first club that has committed the time and resources to forging its own path, and we have the mistakes and lessons to share.

We first went down this path about two years ago when we switched the basic iModules site and CMS to being powered by WordPress. This decision was made prior to my joining the team primarily because the team that did the work already had WordPress experience, and the WordPress backend is fairly intuitive (especially compared to iModules). Despite some developers’ dismissal of PHP and WordPress (including my own), I have grown to appreciate the extensibility of WordPress via actions and filters. We have our own theme, and maintain a couple plugins for our integrations with Stripe and MITAA. I’ll describe these in more detail later.

Once the site and theme were in place, we added the first major feature: event registration. Events are a significant area of programming for many alumni clubs. Events bring together alumni to dine, socialize, be entertained, and be educated. As VP of Arts and Entertainment for the Club of Boston, I spent a lot of time cursing iModules. The user interface for creating/editing events and ticketing is a clunky wizard that is not intuitive to most users. Slow load times mean a prolonged experience for both organizers and registrants. We solved this by switching to Eventbrite which offers a more user-friendly interface for event organizers and attendees.

The Eventbrite integration isn’t perfect. For example, we typically charge non-members a higher ticket price for events; however, Eventbrite doesn’t offer a way for us to signal that a buyer is a club member. We work around this by setting up a discount for every event, and swapping the redirect URL when a member (as detected by WordPress) clicks the registration link. The inability to send more data to Eventbrite means buyers also have to re-input information we already have such as name and email address. Issues aside, we are still happier with this setup than we were with iModules.

Single sign-on

This was the state of the system circa February 2020. As the pandemic ramped up, and more of our events went virtual, we were faced with an interesting challenge. We were now able to open our events to all alumni around the world and offer reciprocal benefits (e.g., MITCOB members pay the same price as MITCNC members), but we had no way of determining club membership outside of our own. Through our partnership with MITAA, we had previously worked on a CAS-based single sign-on (SSO) solution that gave us basic alumni information such as name and email address. To solve the reciprocity problem, we moved to a new OAuth 2.0 authentication provider that gave us access to new APIs. These APIs allowed us to retrieve name, email, club memberships, and graduation year (which will be relevant to memberships mentioned later), enabling us to open events to all alumni worldwide. This new SSO implementation was deployed in late April.

The CAS client was integrated into our WordPress theme, but the OAuth 2.0 client was built as a separate plugin to maintain a separation of concerns, and, eventually, allow other clubs to make use of our efforts if they choose to follow our lead. In fact, pretty much all of the solutions we’ve worked on are available under the MIT license for any interested alumni groups to use. We also welcome merge requests.

Memberships

Many alumni clubs rely on annual member dues to cover operating expenses. iModules’ membership functionality handles this for nearly all clubs, and payments are processed via CyberSource. For a while we have had issues with credit cards expiring and memberships not renewing. Having worked on Stripe Billing, I thought this was odd since card account updaters allow payment processors to request updated card data for expiring cards. We didn’t start digging into this issue in earnest until February when we realized that a significant percentage of recurring revenue was at risk of being lost.

This kicked off discussions with iModules and our payment processor, CyberSource, to identify the root cause of the problem and appropriate solutions. The process was a bit convoluted because MITAA is the account holder, so I was not always able to speak directly with everyone involved. Ultimately we learned that iModules’ implementation of subscriptions and card tokenization is not compatible with any payment processor’s card account updater. Therefore, whether we used CyberSource, or switched to Stripe, we would continue to see lost revenue.

Around this same time we also saw increased complaints with event registration for new club members. Members who joined did not immediately see member pricing for events. Our administrator had to manually verify membership in iModules and register the member in Eventbrite. This was due to data sync delays between MITAA, which owned the API from which we pulled membership information, and iModules, where members joined and paid for membership. That sync took place once nightly, so our WordPress site never had up-to-date membership data.

The combination of falling revenue, poor user experience, and incomplete data meant we had to migrate from iModules and bring memberships “in house” on our own server. You might think I chose to use Stripe because I worked on Stripe Billing, the very product we needed to power memberships. You are correct, however, that was not the only reason. I had limited resources and wanted to minimize the amount of custom code that someone else might need to maintain in the future. Stripe released Checkout in September 2019 to simplify checkout, and I knew that the customer portal was nearing release in June and would be useful to enable members to manage their memberships—change price level, cancel, update payment info. The combination of these two offerings meant we could almost entirely outsource our billing-related UX to Stripe and end up with a much better experience than we had with iModules.

I should note that I did explore other offerings such as WooCommerce, but few had support for both Checkout and customer portal at the time. Additionally, we needed to customize pricing for certain alumni and current students. For example, current students and alumni who graduated in the past year get a year’s free membership. Alumni who graduated 50+ years ago are members of the Cardinal & Gray Society, and eligible for a free lifetime membership. The need for these customizations lead me to create a WordPress Stripe plugin that meets our needs exactly. The use of actions and filters also allow the plugin to be adapted for other use cases beyond those of MIT alumni clubs.

DevOps

We had to go through some growing pains in order to get to where we are today. The most painful moment came when the site went down in mid-February. I did not yet have access to the code repository or any ability to modify production code. Our contractors had all the keys, and they were asleep in Pakistan.

Since that incident we have adopted more devops practices commonly used in industry:

  1. All repositories are now under a common GitLab organization controlled by MITCNC leadership.
  2. The WordPress installation has been rebuilt atop Bedrock to ensure all configuration is tracked in version control.
  3. All changes go through a merge request process that exercises our CI pipeline.
  4. All changes are deployed via CD pipeline rather than FTP.
  5. The production and staging servers are two independent DigitalOcean droplets.
  6. All servers, and development environments, are configured via Ansible (sourced from Trellis) to ensure a consistent, rebuildable, experience across environments.
  7. The production database and uploads are backed up daily. The backups are tested with restores to the staging environment.
  8. Shell access on the servers is only available to those who actually need it; and, those users have their own accounts which are accessible solely via certificates (not passwords).
  9. Access and error logs are shipped to Papertrail.
  10. Exceptions are logged to Sentry, which notifies the team.

All of this has led to a safer, more secure, production environment and deployment experience. We no longer worry about overwriting uncommitted changes that were deployed via FTP. The risk of root credentials being compromised is significantly lower since they are no longer distributed to everyone developing the project. Improved observability has enabled us to set up alerting, and react more quickly to errors in production.

What’s next?

The overall user experience is much better today than it was in February, but it’s still not the best experience. Eventbrite is better than iModules, but the experience is not seamless. I would like to see a solution that is more integrated with our authentication system, and enables members to see their event history. On the backend, we need better APIs for reporting so that we can ingest the data into our data warehouse.

We still rely on iModules for email marketing, and will probably continue to do so. The costs of researching and migrating to a new system outweigh the benefits at this time.

I am working on a few more blog posts related to my work on this project. The original draft of this post included details of our migration to Stripe, but my editor—my wife, Michelle—recommended splitting that into its own post since that content was over half of the draft! I am excited to speak more on the migration because I haven’t seen much content related to payment processor migrations.

I’ve mentioned our data warehouse a couple times in this post, and will go into detail about that in a future post. Additionally, I found very little literature related to WordPress best practices in a devops world, which I think is a shame. The team at Roots has produced some great tools that make WordPress development and deployment a lot better. There are a couple hosting providers that work with it out of the box; however, none get the process exactly right. Expect another post about my journey to find the right hosting provider for our site. Spoiler: you should probably just stick with DigitalOcean, or your preferred raw compute provider, rather than a WordPress-specific hosting service.

If you’re an MIT alumni club leader wanting to learn more, WordPress user interested in devops, Stripe user interested in migrations, or just someone who is curious or wants to chat, get in touch.

P.S. Thank you to everyone at MITAA who helped make this happen. This was certainly a team effort, but I especially want to highlight Joe and David who were instrumental in our SSO integration and Stripe migration!