Project choice

There are many open source projects out there, but it will be most fruitful to survey those which are similar enough to CPython on a couple key metrics:

1. the number of contributors and their activity (there are scaling issues that don’t make the governance models of very small projects very enlightening for our purposes) ;
2. being mostly or partly community-driven (company-driven projects can afford different governance options, since the company hierarchy has power over the main participants) ;
3. being faced with important design decisions that require a somewhat formal decision process.

Key people and their functions

In the Rust project there are teams responsible for certain areas. For language features there is a “lang team”, for tooling there’s “dev tools” and “Cargo”, and so on. Contentious issues have facilitators to drive discussion who often aren’t the decision makers. Typically the facilitators are authors of the proposed changes (see “Controversial decision process” below). They ensure all key decision makers are involved along with interested community members. They push towards an agreeable outcome via iteration.

In practice this means decisions are rarely escalated to the core team.

The most common role of a contributor is team membership. Issue triage/code review privileges without team membership is rare. Contributors have full commit access, code ownership separation is based on trust. Writing to the compiler repository is frowned upon, all changes go through pull requests and get merged by an integration bot after they were reviewed and approved.

New team members are added by nomination by an existing team member.

Regular decision process

Primary work happens via GitHub issues and pull requests. Approving a pull request by any team member allows it to be merged without further process. All merged pull requests end up in the next stable version of Rust.

Notifying relevant people by mentions is important. Listening to the firehose of e-mails for all GitHub activity is not popular.

There are planning and triage meetings open to the public happening on IRC and Discord. They are not very popular because most of work happens through GitHub. Discussions also happen on official Rust forums (https://users.rust-lang.org/ and https://internals.rust-lang.org/). There is a dedicated moderation team responsible for taking notes and enforcing code of conduct.

Controversial decision process

Larger or controversial work goes through a RFC process. It allows everyone to express their thoughts and iterates towards a resolution. At some point when all blocking concerns of relevant team members are addressed, they sign off on the RFC and it reaches a “final comment period”. That does not require consensus amongst all participants, rather there should not be a strong consensus against the proposal.

After 10 days the RFC is merged unless any new blocking concerns are raised by team members. A “merge” signifies that work towards implementing the feature and integrating it can now happen without interruption. An RFC doesn’t have to have a reference implementation for it to be accepted.

The other possible results of the “final comment period” are to:

• postpone the RFC (similar to the Deferred status in PEPs),
• get it back into discussion if blocking concerns can be addressed, or
• close it if blocking concerns are not solvable. When an RFC is marked as closed, there is a 7-day grace period to debate whether it should be closed.

In practice registering concerns with an RFC happens very often initially but rarely causes for the RFC to be entirely killed.

This process scales well for small-contention changes and/or smaller changes. For the largest controversial changes the discussion gets unwieldy. This is a topic currently (as of August 2018) on the minds of the Rust team (see: “Listening and Trust, part 1”, “Listening and Trust, part 2”, “Listening and Trust, part 3”, “Proposal for a staged RFC process”).

Planning a new release

Every six weeks the Rust compiler is released with whatever it contained at the time. There are no LTS channels or releases yet but this concept is planned to make redistributors able to keep up with development better.

Every few years a so-called “Edition” is released. Those are milestone releases with full sets of updated documentation and tooling. They can be backwards incompatible with previous editions. External packages opt into breaking changes in their crate metadata. The Rust compiler supports all editions that existed prior to its release. Linking between crates of any supported edition is possible.

Changes in the process over time

The Rust programming language was started by Graydon Hoare who developed it as a personal project for a few years. When Mozilla started sponsoring the project, the team slowly grew with Graydon as a BDFL-style figure. He left the project in 2013. Rust functions without a BDFL since. The RFC process was put in place later. Initially some design discussions happened during closed-door weekly video meetings which was shut down in May 2015 (before the 1.0 release of Rust), organically replaced with open discussion and direct influence of teams.

The number of teams is growing in time. The number of technical decisions made by the core team is decreasing, instead those get delegated to respective teams.

The concept of a “final comment period” was introduced to encourage more public discussion and enable reacting to a change about to being made, instead of having to revert a rushed decision that was already made.

Key people and their functions

The OpenStack community is made up of many distinct project teams, responsible for producing different components of the software (block storage management, compute management, etc.) or managing different parts of the processes the community follows (such as tracking the release schedule). Each team is led by a Project Team Lead (PTL), elected by the Active Project Contributors for that project.

Active Project Contributors (APCs) are recent contributors to a given project team. APC status formally requires two things: becoming an individual member of the OpenStack Foundation (membership is free) and having a change merged within the last year (two development cycles) in a repository managed by a project team.

The elected PTL serves a term equal to one development cycle (roughly 6 months). There is no restriction on the number of consecutive terms a person may serve as PTL, and it is common for someone to serve for several terms in a row. It is also not unusual for a team to have only one candidate volunteer to serve as PTL for a given cycle, in which case there is no need for an election.

The PTL represents the team in all cases except where they have explicitly delegated some responsibility. For example, many teams designate a separate release liaison to manage the release process for a development cycle. The PTL also serves as a final decision maker in cases where consensus cannot be reached between the team members.

While the APCs all vote for the PTL of a team, in many other cases only the core reviewer team will be consulted on policy decisions for the team. Anyone may review any patch for any OpenStack project. After someone demonstrates that they have a good grasp of the technical issues of a project, that they provide useful feedback on reviews, and that they understand the direction the project is going, they may be invited to become a member of the core review team. Unlike in many other communities, this status does not grant them the right to submit code without having it reviewed. Rather, it asks them to commit to reviewing code written by other contributors, and to participate in team decision-making discussions. Asking someone to become a member of the core review team is a strong indication of trust.

The Technical Committee (TC) is responsible for managing the development of OpenStack as a whole. The 13 members of the Technical Committee are directly elected by APCs from all project teams. Each member serves a term of two development cycles (roughly 1 year), with the elections split so that only about half of the members’ terms expire at any time, to ensure continuity. The TC establishes overall policies, such as the criteria for adding new project teams, the deprecation policy for Python 2, testing requirements, etc.

Regular decision process

All elections for PTL or TC members use https://civs.cs.cornell.edu to run a Condorcet election. This system was selected because it emphasizes consensus candidates over strict popularity.

The OpenStack contributor community relies on 3 primary tools for discussion: the openstack-dev mailing list, a gerrit code review instance at https://review.openstack.org, and a set of OpenStack-specific IRC channels on Freenode. There are a few teams whose contributors are based primarily in China, and they have trouble accessing IRC. Those teams tend to use alternative platforms such as WeChat, instead.

The tool used for discussing any given decision will vary based on its weight and impact. Everyone is encouraged to use either the mailing list or gerrit to support asynchronous discussion across a wider range of timezones and firewalls, especially for publicizing final decisions for the rest of the community.

Policy decisions limited to a single team are usually made by the core review team for a project, and the policies and decision processes may vary between teams. Some groups write down their team policies in their documentation repository, and use the code review tool (gerrit) to vote on them. Some teams discuss policies on IRC, either ad hoc or during a regularly scheduled meeting, and make decisions there. Some teams use the mailing list for those discussions. The PTL for the team is responsible for ensuring the discussion is managed and the outcome is communicated (either by doing so directly or ensuring that the task is delegated to someone else).

All team policy decisions need to be compatible with the overall policies set by the Technical Committee. Because the TC tends to make broader governance decisions that apply to the entire contributor community, the process for discussing and voting on those decisions is described more formally, including specifying the number of votes needed to pass and the minimum length of time required for discussion. For example, most motions require 1/3 of the members (5) to pass and must stay open at least 3 days after receiving sufficient votes to pass, ensuring that there is time for dissent to be registered. See the Technical Committee Charter and house rules for more details.

Significant design decisions are usually discussed by reviewing a specification document, somewhat similar to a PEP, that covers the requirements, alternatives, and implementation details. Feedback is solicited from all contributors, and then specifications are eventually approved or rejected by members of the core review team for a project. Some teams require only 2 reviewers to approve a design, while other teams require a stronger indication of consensus before a design is approved. Each team sets a deadline for approving specifications within each development cycle, to encourage contributors to work out designs for significant new features early and avoid risk from changes late in the cycle.

Smaller technical decisions are typically made by reviewing the patch(es) needed to implement the change. Anyone may review any patch and provide technical feedback, but ultimately two core reviewers for a team are needed to approve most changes (exceptions are often made for trivial changes such as typos or for fixes that unblock the CI gating system).

Controversial decision process

Controversial, or merely complicated, decisions frequently expand outside of specification reviews to mailing list discussions. They often also result in discussions at one of the regularly scheduled in-person community gatherings. Because many members of the community cannot attend these events, the discussions are summarized and final decisions are made using on-line tools as much as possible.

The PTL is responsible for deciding when consensus has been reached for decisions that affect a single team, and to make a final call in rare cases where consensus has not been reached and a decision absolutely needs to be made. The TC acts as a similar decision-making group of last resort for cases where issues between teams cannot be resolved in another way. Such escalation of decision-making ends up being rarely necessary, because the contributors directly involved generally prefer to come to a consensual agreement rather than escalate the decision to others.

Planning a new release

OpenStack has a major release about every 6 months. These are coordinated date-based releases, which include the work finished up to that point in time in all of the member projects. Some project teams release more often than every 6 months (this is especially true for teams building libraries consumed by other teams). Those smaller releases tend to be produced when there is content (new features or bug fixes) to justify them.

The schedule for each development cycle, with deadlines and a final release date, is proposed by the release management team, in coordination with the Foundation staff (releases are generally aligned with the calendar of in-person events), and then the community has an opportunity to provide feedback before the final dates are set.

Decisions about priorities for each development cycle are made at the team level and the TC level. Core review teams prioritize internal work, such as fixing bugs and implementing new features. The TC selects community goals, which usually require some amount of work from all teams. Agreeing to these priorities at the start of each cycle helps the teams coordinate their work, which is especially important because the implementation will require reviews from multiple team members.

Changes in the process over time

Over the last 8 years the number of OpenStack project teams has grown from 2 to 63. The makeup of the Technical Committee has changed to accommodate that growth. Originally the TC was made up of PTLs, but as the membership grew it became impractical for the group to function effectively.

The community also used to be organized around “program areas” rather than project teams. A program area covered a feature set, such as gathering telemetry or managing block storage. This organization failed when multiple teams of people wanted to work on the same feature set using different solutions. Organizing teams around the code they deliver allows different teams to have different interpretations of the same requirements. For example, there are now several teams working on different deployment tools.

Key people and their functions

The key people in Jupyter’s Governance are the BDFL, Fernando Perez, and the Steering Council. Contributors can be given a special status of core contributor. Some may also be Institutional Contributors, who are individuals who contribute to the project as part of their official duties at an Institutional Partner.

Fernando Perez, the project founder, is the current and first BDFL. The BDFL may serve as long as desired. The BDFL succession plan is described in the Main Governance Document. In summary, the BDFL may appoint the next BDFL. As a courtesy, it is expected that the BDFL will consult with the Steering Council. In the event that the BDFL can not appoint a successor, the Steering Council will recommend one.

Core contributors are individuals who are given rights, such as commit privileges, to act in the best interest of the project within their area of expertise or subproject. An existing core contributor typically recommends someone be given core contributor rights by gathering consensus from project leads, who are experienced core contributors as listed in the README of the project repo.

To be recommended and invited as a Steering Council member, an individual must be a Project Contributor who has produced contributions that are substantial in quality and quantity, and sustained over at least one year. Potential Council Members are nominated by existing Council members and voted upon by the existing Council after asking if the potential Member is interested and willing to serve in that capacity.

Regular decision process

Project Jupyter is made up of a number of GitHub organizations and subprojects within those organizations. Primary work happens via GitHub issues and pull requests. Approving a pull request by any team member allows it to be merged without further process. All merged pull requests end up in the next stable release of a subproject.

There is a weekly, public Project-wide meeting that is recorded and posted on YouTube. Some larger GitHub organizations, which are subprojects of Project Jupyter, e.g. JupyterLab and JupyterHub, may have additional public team meetings on a weekly or monthly schedule. Discussions occur on Gitter, the Jupyter mailing list, and most frequently an open issue and/or pull request on GitHub.

Controversial decision process

The foundations of Project Jupyter’s governance are:

• Openness & Transparency
• Active Contribution
• Institutional Neutrality

During the everyday project activities, Steering Council members participate in all discussions, code review and other project activities as peers with all other Contributors and the Community. In these everyday activities, Council Members do not have any special power or privilege through their membership on the Council. However, it is expected that because of the quality and quantity of their contributions and their expert knowledge of the Project Software and Services that Council Members will provide useful guidance, both technical and in terms of project direction, to potentially less experienced contributors.

For controversial issues, the contributor community works together to refine potential solutions, iterate as necessary, and build consensus by sharing information and views constructively and openly. The Steering Council may make decisions when regular community discussion doesn’t produce consensus on an issue in a reasonable time frame.

Voting

Rarely, if ever, is voting done for technical decisions.

For other Project issues, the Steering Council may call for a vote for a decision via a Governance PR or email proposal. Acceptance requires a minimum of 80% of the Steering Council to vote and at least 2/3 of the vote must be positive.

The BDFL can act alone to accept or reject changes or override the Steering Council decision; though this would be an extremely rare event. As Benevolent, the BDFL, in practice chooses to defer that authority to the consensus of the community discussion channels and the Steering Council.

Planning releases

Since Project Jupyter has a number of projects, not just a single project, the release planning is largely driven by the core contributors of a project.

Changes in the process over time

The process has remained consistent over time, and the approach has served us well. Moving forward The Project leadership will consist of a BDFL and Steering Council. This governance model was a formalization of what the Project was doing (prior to 2015 when the Main Governance Document was adopted by the Steering Council), rather than a change in direction.

Key people and their functions

The project recognizes three kinds of contributors. Members of the core team, the Technical Board, and Fellows. Regular core committers no longer exercise their “commit bit”, instead they rely on pull requests being reviewed and accepted. The Technical Board steers technical choices. Fellows are hired contractors who triage new tickets, review and merge patches from the committers and community, including non-trivial ones.

Core team members are added by nomination and vote within the core team, with technical board veto (so far not exercised). Technical board is elected by and from the core team membership every 18 months (every major Django release). Sub-teams within the core team are self-selected by interest.

Regular decision process

Most day-to-day decisions are made by Fellows and sometimes other active core team members.

The core team votes on new members which requires a 4/5 majority of votes cast, no quorum requirement. The Technical Board has veto power. This power was never exercised

Controversial decision process

The Technical Board occasionally approves Django Enhancement Proposals (DEPs) but those are rare. The DEP process is roughly modeled after PEPs and documented in DEP 1. DEPs are mostly used to design major new features, but also for information on general guidelines and process.

An idea for a DEP should be first publicly vetted on the django-developers mailing list. After it was roughly validated, the author forms a team with three roles:

• authors who write the DEP and steers the discussion;
• implementers who prepare the implementation of the DEP;
• a shepherd who is a core developer and will be the primary reviewer of the DEP.

The DEP’s draft is submitted, assigned a number, and discussed. Authors collect feedback and steer discussion as they see fit. Suggested venues to avoid endless open-ended discussions are: separate mailing lists, Wiki pages, working off of pull requests on the DEP.

Once the feedback round is over, the shepherd asks the Technical Board for review and pronouncement. The Board can rule on a DEP as a team or designate one member to review and decide.

In any case where consensus can’t be reached, the Technical Board has final say. This was never exercised.

Planning a new release

Releases are done on a fixed time-based schedule, with a major version every 18 months. With paid Fellows to ensure the necessary work gets down, on-time releases are routine.

Changes in the process over time

Django originally had two BDFLs: Jacob Kaplan-Moss and Adrian Holovaty. They retired (Adrian’s post, Jacob’s post) 9 years into the project’s history. Following the stepping down, the DEP process was defined.

Key people and their functions

There is a formal design team and a release management team working at Microsoft. The main person behind the project is currently Anders Hejlsberg as some of the original members of the team have left the company.

Regular decision process

Microsoft, where the project is developed, has a strong planning culture so development roadmaps are released long in advanced, notes from design discussions held at Microsoft get published quickly and meetings are sometimes broadcast using Skype.

External contributions are encouraged through pull requests on GitHub. Suggestions for new use cases or features are given by issues on GitHub. This serves like an ad-hoc PEP-like process. There is some discussion over social media (Twitter) as well.

Controversial decision process

Hejlsberg is the central figure of the project in terms of language design, synthesizing community needs into a cohesive whole. There is no formal process to externally contribute to the design of the language.

The TypeScript team filters through and integrates community suggestions. The main advantages of this setup are that there is strong and consistent design with dependable scheduling and execution. While there is transparency of intentions and plans, the disadvantage of this model is that community involvement is limited to pull requests and suggestions.

Planning a new release

Microsoft determines the release schedule, communicates dates and features well in advance. Nightly builds are usually stable (with a significant portion of users on this release form).

Versioned releases are done every 1 - 3 months, with a roadmap available on GitHub.

Changes in the process over time

TypeScript is likely the first notable project by Microsoft developed fully in the open (versus source-available).

Open-sourcing of TypeScript by Microsoft was a planned feature from the inception of the project. Before the first open release was made, the language was driven fully by needs identified by the original teams and the early in-house users. The initial open-sourcing happened via the now-defunct Microsoft CodePlex platform. It didn’t have a well-defined routine of accepting external contributions. Community engagement rose significantly after the project got moved.

Key people and their functions

The Astropy Project team’s responsibilities are spread over many different roles [1], though frequently a person will have several roles.

The main body overseeing the Astropy Project is the Astropy Coordination Committee (CoCo) . Its key roles are dealing with any financial issues, approving new packages wanting to join the Astropy Project, approving or rejecting Astropy Proposals for Enhancement (APEs) [2], and generally anything that’s “leadership”-oriented or time-sensitive. As of this writing, the committee has four members, and might grow or shrink as the demands on the committee change.

Regular decision process

Controversial decision process

Simpler controversial decisions are generally discussed on the astropy-dev mailing list [3], and after a reasonable time either there is a clear consensus/compromise (this happens most of the time), or the CoCo makes a decision to avoid stalling.

More complicated decisions follow the APE process, which is modeled after the PEP process. Here, the CoCo makes the final decision after a discussion period, open to everyone, has passed. Generally the CoCo would follow the consensus or majority will.

Planning a new release

The major release timing is on a fixed schedule (every 6 months); whatever is in at that time goes in.

Changes in the process over time

The CoCo and the “Open Development” ethos came from the inception of the Project after a series of votes by interested Python-oriented astronomers and allied software engineers. The core results of that initial discussion were embodied in the Vision for Astropy document [4].

The existence of the formal roles and most of the rest of the above came as evolutionary steps as the community grew larger, each following either the APE process, or the regular process of a proposal being brought up for discussion and vote in astropy-dev [3]. In general all evolved as sort of ratification of already-existing practices, only after they were first tested in the wild.

Self-appreciation

The fact that anyone who has the time can step in and suggest something (usually via PR) or vote on their preference, leads to a sense that “we are all in this together”, leading to better-coordinated effort.

Additionally, the function of the CoCo as mostly a tie-breaking body means that there’s no sense of a dictator who enforces their will, while still giving clear points of contact for external organizations that are leery of fully-democratic governance models.

References

Key people and their functions

Microsoft has a hierarchy that ultimately reports to the CEO. Below the CEO are a number of organisations, some of which are focused on engineering projects (as opposed to sales, marketing or other functions). These engineering organisations roughly break down into significant product families - for example, there has been a “Windows group”, an “Xbox group”, and a “server and tools group”. These are typically led by Executive Vice Presidents (EVPs), who report to the CEO.

Below each EVP are many Corporate Vice Presidents (CVPs), each of which is responsible for one or more products. This level is where the hierarchy becomes relevant for the purposes of this PEP - the CEO and EVPs are rarely involved in most decision processes, but set the direction under which CVPs make decisions.

Each product under a CVP has a team consisting of Program Managers (PMs) and Engineering Managers. Engineering Managers have teams of engineers who are largely uninvolved in decision making, though may be used as specialists in some cases. For the rest of this section, Engineering refers to anyone from the engineering team who is contributing with a technical-focus, and PM refers to anyone from the program management team contributing with a customer-focus. After decisions are made, Engineering does the implementation and testing work, and PM validates with users that their problem has been solved.

(This is actually a huge simplification, to the point where some people in these roles are offended by this characterisation. In reality, most people in PM or Engineering do work that crosses the boundary between the two roles, and so they should be treated as a term describing the work that somebody is doing in the moment, rather than an identifier or restriction for a person.)

Teams generally represent a feature, while the CVP represents a product. For example, Visual Studio Code has a CVP who is ultimately responsible for decisions about that product and its overall direction (in the context set by their EVP). But many teams contribute features into Visual Studio Code.

For complete clarity, the CEO, EVPs, and CVPs do not ever directly modify source code. Their entire role is to provide direction for whoever is immediately below them and to adjudicate on controversial decisions.

Regular decision process

Changes to product code that are not visible to external users are made solely by Engineering. Individual engineers will be assigned tasks by a designated engineering manager, or may self-assign. Promotion to increasingly senior positions generally reflects trust in the individual’s decision-making ability, and more senior engineers are trusted to make decisions with less validation from the rest of the team. Most bugs are covered by this process (that is, fixing a user-visible problem without changing the intended experience is an Engineering decision).

Decisions affecting users of a particular feature are made by the PM team for that feature. They will use whatever data sources available to identify an issue, experiment with alternatives, and ultimately prepare a design document. Senior members from PM and Engineering will review designs to clarify the details, and ultimately an artifact is created that the feature team agrees on. Engineering will use this artifact to implement the work, and PM will later use this artifact to validate that the original issue has been resolved.

Senior members of Engineering and PM teams for a feature are expected to make decisions in the spirit of the direction set by their CVP. Teams have regular meetings with their CVP to discuss recent decisions and ensure consistency. Decisions that are not obviously in line with CVP expectations are escalated to the controversial process.

Controversial decision process

When decisions require cross-team coordination, or do not obviously align with previous CVP guidance, teams will escalate decision making. These often include decisions that involve changing direction, attempting to reach a new or different group of users, deprecating and removing significant features (or on a short timeframe), or changes that require quick releases.

In general, CVPs are not intimately familiar with all aspects of the feature team’s work. As a result, the feature team must provide both a recommendation and sufficient context for the decision that the CVP can decide without additional knowledge. Most of the time, the first attempt results in a series of questions from the CVP, which the team will research, answer and attempt the decision again at a later date.

Common questions asked by CVPs are:

• how many users are affected by this decision?
• what is the plan for minimizing impact on current users?
• how will the change be “sold”/described to potential users?

CVPs are expected to have a strong understanding of the entire field, so that they can evaluate some questions for themselves, such as:

• what similar decisions have been made by other projects within Microsoft?
• what other projects have plans that may impact this decision?
• what similar decisions have been made by projects outside Microsoft?
• do users need it?
• is it in line with the direction set by their EVP?

Decisions made by CVPs are generally arbitrary and final, though they typically will provide their rationale.

Planning a new release

Releases involve coordinating a number of feature teams, and so rarely attempt to include input from all teams. A schedule will be determined based on broader ecosystem needs, such as planned events/conferences or opportunities to take advantage of media attention.

Teams are informed of the release date, the theme of the release, and make their own plans around it following the above decision making process. Changing the release date is considered a controversial decision.