Enterprise-grade GitHub: How Azure scaled to 2,000+ employees on GitHub
As Azure expanded its open source presence over the last four years, we’ve had to rethink our processes and minimize the manual steps involved in running a GitHub organization at scale. We’ve automated large org management with a web app powered by a few Azure services (including App Service, Active Directory and Azure Storage), the GitHub API, great open source, and contributed to OSS ourselves along the way.
In this post I’ll introduce our open source portal for GitHub that is now published on GitHub. I’ll share the features, service building blocks, and end with some perspective on how we’ve grown on GitHub, ending with pointers to more info. I’ve also prepared a detailed post covering this project in additional detail from its hackathon roots through to where it is today.
Developers with organizations of any size can easily benefit from this portal, leveraging Azure Active Directory to bring self-service GitHub tooling to your engineering group.
Azure Open Source Portal for GitHub
Joining an org on GitHub requires an invitation to be sent from an org administrator to a specific GitHub username – a username which often does not correlate with a corporate alias or e-mail address. GitHub is an awesome product and service, especially for small and medium-sized teams: a strong emphasis on simplicity works for the majority of scenarios. We’ve just outgrown the simple and needed to automate to keep our heads above water and focused on our other important work.
Although GitHub Enterprise supports some more integration points, when it comes to GitHub.com, where we collaborate with the open source world, everyone needs an official GitHub account today.
The Azure Open Source Portal for GitHub started as a hackathon project: a challenge to improve the manual maintenance problem in a new way. By allowing users to authenticate with both their GitHub identity and corporate identity, and automating the process of joining, we can remove a big bottleneck from the standard GitHub workflow, enforce simple open source policies, and keep our lawyers happy.
Over time we’ve adapted alongside GitHub’s improvements to their organization model (a big update just happened last month) and integrated with other systems at the company. Microsoft as a whole has a very similar process.
These self-service tools and simple workflows allow our engineers to be productive at their own pace, delegating decisions to team maintainers (scoped to the project, rather than the admins).
Although it’s been fun to adapt the portal over time in parallel with improvements GitHub has made, in time we hope to be able to retire many parts of the portal as GitHub iterates. They’ve been a very good partner to us and we’re excited to continue to grow alongside.
Let’s take a look at some of the simple scenarios and features in the portal…
Instantaneous onboarding experience
As an org grows in size, it isn’t ideal to scale up the admins count. Admins are given global read and write privilege, full access to the danger zone for repos, teams and the organization itself, and likely have more important things to focus on than inviting new members to the org by hand. With scale comes important questions: “Is this username a Microsoft employee? Do we know who this person is? Does this employee still work with us, or did she leave for another company?”
The portal uses the GitHub API to authorize operations on behalf of admin accounts. With the user’s permission via OAuth, we can send and accept their invitation immediately, check they have two-factor authentication turned on (helping to drive us to 100% compliance), and then offer them the opportunity to join teams they might need to work with.
The portal also gives us a chance to share links to our open source training, best practices, and build a rich experience. Some of our engineers are experiencing GitHub for the first time as they onboard, so these are important moments.
Support for multiple organizations
The portal supports many organizations. Users can join one or many. We can make sure to remove former employees from all of our organizations when changes happen. The portal is configured so each organization can have independent and unique underlying credentials. Organization admins need not be admins of all the orgs that the portal manages.
Role-based organization administrators
Azure’s family of GitHub orgs do not have regular employee users as admins, but instead special protected role-based admin accounts. This reduces the risk for mistakes by admins, such as accidentally pushing commits to a repo. Each organization has a special “sudoer” team that gives specific individuals sudo rights for that org to help with vacation coverage and urgent requests.
Presenting employee identities alongside GitHub usernames
Inside the portal users’ links (employee and GitHub identities) are shown alongside for all operations: adding members and maintainers, during permission requests to join teams, and when requesting a new repo. This makes it easy for team members to add their co-workers without hunting down a list of known GitHub usernames.
There are also some web hook integrations and REST APIs exposed by the app that enable other open source tooling at Microsoft to interface with the data store including the Azure CLA bot.
Delegation to team maintainers
We trust our engineers. Instead of leaving decisions to global admins, decisions are delegated to team maintainers. Maintainers own their permission requests. Each org has a special team assigned to approve new repos.
Elevated capabilities given to team maintainers include the ability to take their repos public, add additional employees to their teams, as well as manage any one-off permissions, web hooks or outside collaborators for their repos.
Prior to October 2015, our portal had its own implementation of “team owners” who were given the ability to make permission and repo decisions inside the portal. With the October GitHub org improvements, we’ve locked to their newer model and updated our systems.
Requests assigned via GitHub issues
Instead of using a database to manage the permission and repository create workflows, we use a lightweight NoSQL table design coupled with GitHub issues themselves. When a request comes in to join a team, an issue is opened in a private GitHub repo that all members have access to, the request is mentioned to all the GitHub users who maintain the team, and a link is provided to take action on the request. Using GitHub issues adds a level of transparency and simplicity.
Powered by Azure’s cloud infrastructure
The portal leverages OSS and makes use of Azure services including:
- App hosting, availability, continuous deployment and configuration using Azure App Service. CI/CD is configured to make dev/test loop easy and responsive. Deployment slots allow the portal’s upcoming changes to be flighted using a staging Git branch ahead of shipping. In a pinch, the real-time Diagnostic Log Stream can make diagnosing issues quick and painless, and Application Settings can set up a service alert banner in case a known issue in an API or the portal itself is discovered.
- By using Azure Active Directory and the Microsoft tenant, our application is able to authenticate and authorize employees.
- The virtual links for users are maintained in Azure Table Storage, part of the storage service. The highly available, geo-redundant data store is quick and inexpensive.
- Azure Redis Cache is a fully managed cache used to help reduce pressure on the GitHub API, store sessions and other key data in flight. This lets us deploy the app in as many regions or instances as needed while sharing the cache.
- Visual Studio Application Insights and its Node.js npm module are used to monitor both web client performance, server-side response times, and perform availability tests from around the world.
Here’s a look at the Azure portal. Key stats and information about the portal have been pinned to the start page, including an app insights timeline view and response time charts, Redis get/set statistics over time, plus quick access and information to the other Azure services and App Service deployment currently in prod.
Great open source
Front-end components used in the project include Bootstrap, jQuery, and several small jQuery plugins. The portal is a Node.js app which uses the following NPM modules: applicationinsights, async, azure-storage, express, jade, moment, node-uuid, octonode, passport, passport-azure-ad, passport-github, redis. The app has been developed across Mac, Windows and Linux.
By being able to check out the source and even contribute to this portal, I hope others will learn from our approach and provide feedback. Azure is also coordinating with Microsoft’s open source program office to share this portal as part of an initiative around corporate open source tooling.
Azure on GitHub: four years later
When we set up our first GitHub organization in late 2011, we were accelerating toward a set of deliverables for June 2012 and GitHub was a part of that: a place to demonstrate a new openness beyond the typical Microsoft stack. We pulled out a corporate Amex to pay GitHub, immediately appointed 20 people to be org admins, and it all grew from there. Instead of developing projects “in the dark”, we were able to work with our legal team and find a way to build many of our approved OSS projects right there on GitHub. Early feedback, including community code reviews, has really been a positive change, and with so many Azure customers using GitHub, it’s been a lot of fun.
Today we have over 2,000 members in our GitHub org from all corners of Microsoft, just a few owner accounts, a formal PO process with GitHub sales, a great two-factor participation rate in the org, and many engineers enjoying our self-service capabilities… all with decisions delegated to team maintainers who are close to the work. We have engineering teams collaborating with the community on everything from resource templates (deploy a highly available MongoDB cluster to the cloud straight from GitHub) to improving the documentation on the Azure web site (contribute to our Linux on Azure tutorial), SDKs and samples. It will be fun to see what’s next.
Open Source Portal for GitHub
- Detailed blog post about the OSS portal story, architecture and journey by Jeff Wilcox
- Clone, fork and check out the Azure Open Source Portal for GitHub on GitHub
- This is also a great time to read about the Azure engineering team’s journey to Git source control inside Microsoft
Node.js on Azure
- Follow a tutorial: creating a Node.js web app in Azure App Service
- Learn about using Azure table storage and Node.js together
- Check out the Azure SDK for Node.js and Azure Storage SDK for Node.js on GitHub
Azure Services and cloud infrastructure
- Azure App Service: Build and host intelligent web and mobile apps
- Azure Active Directory: Identity and access management for the cloud
- Azure Redis Cache: Dedicated Redis managed by Microsoft
- Azure Storage: Blog, table, queue and other essential data services
- Visual Studio Application Insights: Detect issues, diagnose crashes and track use of your web apps
- If you’re new to Azure, check out the Azure free trial.
We’re hiring: Would you like to work at the “new Microsoft” as it embraces open source technologies? The Azure team is hiring great engineers. You can take a look at some of our open positions and learn about working at Microsoft on our career site.
Source: Microsoft Azure News