tl;dr We interviewed developers in Eclipse, R/CRAN, and Node.js/npm about how they distribute the costs of breaking change. Community values, practices, and tools differ significantly among ecosystems. With Eclipse valuing stability, R/CRAN valuing ease and rapid access for end users, and Node.js/npm valuing ease for developers, they each adopted distinct practices and policies. Understanding values can help resolve conflicts and to design appropriate policies and tools. We've also recently conducted a survey to learn about values and practices in 18 ecosystems, which we report separately in our survey results.
Change in software ecosystems with interdependent but independently developed packages can be disruptive. Breaking changes (e.g., removing, renaming, changing contracts) can ripple through the entire ecosystem, causing rework for many package maintainers and users.
Developers tend to avoid breaking changes when they can, but they are still common in practice. Reasons for breaking changes are similar across ecosystems, primarily: new features, changed requirements, removing technical debt, improving performance, and fixing bugs.
While performing the change can be easy, a change can cause huge costs in rework and interruptions for users of the package. As each participant has their own goals, priorities, habits, and rhythms, there is often (implicit) negotiation about changes regarding when, how, and who. Different developers and different communities have different attitudes toward who should pay the costs of a change and when.
Freely made changes can impose high and unpredictable costs on users. Making a change is easy but can cause significant rework costs downstream.
Forgoing needed changes can be costly for the package maintainer (technical debt, opportunity costs).
Maintainers have a variety of strategies for taking on some extra work, in order to reduce rework costs and interruptions for users.
Maintainers can reduce rework costs for users by investing extra work when making the change:
Maintainers can invest extra work to reduce interruptions to users, delay their rework costs, and help them plan ahead:
Third parties or volunteers can take on some of change costs by vetting changes or mediating between parties. Acting as gatekeepers, they may review changes for quality and security or curate a selection of recommended packages. Some may choose to release a whole collection of packages at once in regular intervals.
Users can reduce their own exposure to change and manage their effort in monitoring change:
Common practices and policies regarding breaking changes differ drastically among different ecosystems. These differences can be explained by community values, which again align with power structures.
The Eclipse foundation publishes more than 250 open source projects. Its flagship project is the Eclipse IDE, created in 2001. The IDE is built from the ground up around a plugin architecture. Projects can apply to join the Eclipse foundation through an incubation process in which their project and practices come under the Eclipse management umbrella. It is also common practice to develop both commercial and open-source packages separately from the foundation, and publish them in a common format on a third-party server and register them in the "Eclipse marketplace", from which over 1600 external Eclipse packages that can be installed from third-party servers through a GUI dialog.
A core value of the Eclipse community is backward compatibility. This value is evident in many policies, such as
API Prime Directive: When evolving the Component API from release to release, do not break existing Clients. One can often expect that packages (e.g., commercial business solutions) developed 10 years ago will still work in a current Eclipse revision without modification. The value aligns with business needs of corporate sponsors, who provide significant funding.
Expensive for changer
Willing to accept high costs and opportunity costs for compatibility
Community created educational material explaining stability policy and nuances of binary compatibility.
Detailed change documentation for each release.
Amounting technical debt: deprecating methods rather than removing, deprecated methods kept for > 10 years, suggesting workarounds for compatible extensions (e.g., IDetailPane2, IDetailPane3); many changes and code modernization not possible.
Static analysis tool API tools to check compatibility before release broadly adopted (tools enforce policy and ease compliance).
Coordinated yearly releases, synchronization among all core projects (high overhead, causes some friction).
Backporting uncommon, since compatibility allows easy updates.
Minimal rework and interruption costs for users
Dependencies reliable and stable; involvement and rework rarely required.
Convenient for vendors who can support latest release and older ones without changes (10 year old plugins still work).
Yearly scheduled updates provide planning horizon, sufficient for many.
Perceived lack of modernization, slow decision making, and stagnant development that discourages new developers. Synchronized releases cause significant overhead and some conflict. (Disclaimer: Oversimplification, but consistent with interviewees views)
Typically, if you have hip things, then you get also people who create new APIs on top ... to create the next graphical editing framework or to build more efficient text editors. ... And these things don’t happen on the Eclipse platform anymore.
The Comprehensive R Archive Network (CRAN) has managed and distributed packages written in the R language since 1997. R is an interpreted language designed for statistics. R has multiple repositories with different policies and expectations, including Bioconductor and R-Forge; we focus on CRAN, the largest one. CRAN formally exists under the umbrella of the R Foundation, but sets its own policies.
CRAN contains over 8000 packages. Of these, 14 are included along with the R source code, and 15 more are considered "recommended" and are bundled in binary installs. About 2200 more are cataloged as official suggestions for 33 different specializations such as finance and medical imaging. While users can distribute R scripts in many forms, publishing a package on CRAN has benefits to visibility and making it easy to install. Installing packages from CRAN is automated in every installation of the R language.
A core value of the R/CRAN community is to make it easy for end users to install and update packages. Although not explicitly represented in policy documents, this value was apparent from many interviews; for example
CRAN primarily has the academic users in mind, who want timely access to current research. CRAN is designed and evolved by team of volunteers concerned about end users.
Encouraged to work with downstream maintainers
CRAN pursues snapshot consistency: at every point in time, the newest version of every package should be compatible with the newest version of every other package in the repository.
No policy against breaking changes if snapshot consistency is maintained.
Changer needs to reach out to affected users to coordinate change; volunteers might mediate.
Developers are encouraged to run tests of all downstream packages before release.
No central or scheduled releases (incompatible with value of timely access), no parallel releases, little emphasis on version numbering (only latest version matters).
Conflict-free updates for end users, but timely rework obligation for package maintainers
Convenient package installation for package users.
No easy mechanism to install and depend on old versions though.
Urgency for package maintainers affected by breaking changes: packages may be archived if not updated within few weeks.
Aggressive reduction of dependencies by package maintainers to reduce exposure to urgent change, sometimes prefering code cloning over reuse.
Typically waiting for emails to learn about breaking changes in dependencies; active monitoring not common.
Volunteers check packages and ensure consistency
The CRAN team vets each package revision submitted to CRAN.
Partially-automated analysis assists the vetting of revisions: Ensuring consistency by executing tests of all packages that import the changed package within CRAN.
Tooling to check compatiblity with latest version of the R language itself.
The CRAN team may send emails to affected package maintainers.
Some developers welcome the expected coordination and collaboration, but others see it and the urgency produced as a burden, causing constant interruption. Concerns that updates and difficulty to access prior versions can threaten scientific reproducibility. Gatekeeping is expensive but usually accepted by developers due to added value and because of gained prestige of being in repository.
And then I need to [react to] some change ... and it might be a relatively short timeline of two weeks or a month. And that's difficult for me to deal with, because I try to sort of focus one project for a couple weeks at a time so I can remain productive.
The Node.js/npm platform has the somewhat unusual characteristic that multiple revisions of a package can coexist within the same project. That is, a user can use two packages that each require a different revision of a third package without causing a conflict. In that case, npm will install both revisions in distinct places and each package will use a different implementation.
A core value of the Node.js/npm community is to make it easy and fast for developers to publish and use packages. In addition, the community is open to rapid change and has culture of experimenting with APIs until they are right. There is significant emphasis on tool building and community, often through grassroots movements.
Npm was designed by one individual developer for other developers; ease for developers was one of the principles motivating the designer of npm (e.g., as he discusses in the changelog podcast). Therefore, npm explicitly does not act as a gatekeeper; it does not have review or testing requirements; in fact the npm repository contains a large number of test or stub packages. The focus on convenience for developers (instead of end users) was apparent throughout our interviews.
Releasing updates is easy and fast
Breaking changes are generally seen as necessary for progress and are accepted if signaled with increased major release number (semver).
More common to remove technical debt, fix APIs.
Maintenance releases (backporting) common if many users depend on old version, especially for security patches.
No central release planning, no gatekeeping (both would contradict the community's values).
Relatively high frequency of updates requires frequent rework
Npm allows to use multiple versions of a package at the same time (only node version itself can cause conflicts). Therefore, it is easy to keep using old revision: updating is a choice, not an urgency.
Social mechanisms and tools (e.g., greenkeeper.io) help to monitor changes.
Semantic versioning is increasingly reliable and expected by community.
Perceived challenge to find the right balance between progress and burden on users. Progress and ability to chose to not update seen as benefits, but not updating can quickly lead to a large backlog that can require expensive rework to catch up.
Last week's tutorial is out of date today.
Common practices and policies regarding breaking changes differ drastically among different ecosystems, but are typically consistent within. Despite their differences, the practices and policies make sense in each ecosystem considering its community values.
Negotiating change is a difficult problem in software engineering that can easily lead to conflicts among developers with different goals or rhythms. While technical considerations certainly play a role, choices about allocating costs and benefits are fundamentally political decisions. Each analyzed ecosystem has different practices, policies, and tools that align with community values in that ecosystem and achieves a different cost distribution compatible with those values. Community values can be somewhat difficult to distill from the outside -- making community values and the involved tradeoffs explicit and transparent can help to ensure that all stakeholders understand the tradeoffs of decisions made by the platform and the accepted consequences, such as higher costs for certain stakeholders or reduced attractiveness to newcomers. Such political transparency can help to understand and resolve conflicts and to guide design discussions.
All three ecosystems put a lot of effort and trust into tools, which enforce policies or reduce the cost of compliance. Furthermore, the impact of changes can often be reduced if users are included in planning the change so they can influence timing, help limit the disruption to current and future use, and identify specific kinds of help or advice they need. Some dedicated communication channel, such as mailing lists, IRC, and issue trackers, can be helpful as a central place for raising awareness of upcoming changes and for providing an opportunity for input.
There is a large design space in how to build an ecosystem and how to allocate costs among the various stakeholders, but we believe that ecosystems are rarely designed explicitly. Only few developers understand mechanics and their intentions of multiple ecosystems well. Blindly copying practices or adopting tools without understanding values might not be effective. Misalignment might explain less successful ecosystems. Understanding mechanisms and values could be a strong tool to debug a community and design healthier ones, at least with respect to change.
Beyond Eclipse, R/CRAN, and Node.js/npm: What are the values of the PHP, Go, Haskell, or Perl community? What practices for shifting or delaying costs of breaking changes are common in TeX, Rust, or Python? What change-related friction do developers experience on Maven or DockerHub? Do npm-inspired ecosystems share npm's community values? What tools have been adopted in the Swift or Racket community? Do industry-backed communities strive toward stability? We recently conducted a survey about these practices; we will update this web page soon with results.
Most interviewees stated that they avoid breaking changes whenever possible. Almost all interviewees stated that they started with a presumption not to make changes that would affect downstream users, when they could avoid it. Reasons included looking out for their users' best interests and knowing that costs to affected users would come back to them, as users ask for help adapting to the change, ask for the change to be reverted, or seek alternative packages. Two interviewees specifically mentioned concern for downstream users' research (
We're improving the method, but results might change, so that's also worrying - it makes it hard to do reproducible research).
Our interviewees' concern for impacts on users was amplified by the size and visibility of the user base, and the perceived importance and appropriateness of their usage. In fact, many interviewees across all ecosystems had an awareness of their user community and were concerned specifically about the number of users affected and the quantity of complaints that a change would imply, e.g.,
Sometimes you want to rename a function or class, ... However because this would break scripts or packages assuming the old name, you often end up supporting both names. Some interviewees noted that their sensitivity toward avoiding breaking changes grew with experience and with a growing user base, as they learned from feedback received about earlier breaking changes.
Only a few developers were not particularly worried about breaking changes. Some had a strong tie to their users and felt they could help them individually (
We try to avoid breaking their code - but it's easy to update their code). One interviewee expressed an "out of sight, out of mind" attitude:
Unfortunately, if someone suffers and then silently does not know how to reach me or contact me or something, yeah that's bad but that suffering person is sort of [the tree] in the woods that falls and doesn't make a sound as far as I'm concerned.
Finally, there was some debate about bug-fixing changes (see above). While some developers aimed to support downstream users who relied on incorrect behavior, other developers were less concerned when they considered usage as inappropriate, as stated for example:
After upgrading the parser some people complained that their script was no longer working. But the problem was that their syntax was invalid to begin with. It's obviously their fault.
Despite a strong general preference for avoiding breaking changes, there are many cases where the opportunity costs of not making a change are too high. Our interviewees identified several different strategies for how they, as package maintainers, routinely invest effort to reduce or delay the impact from their changes for downstream users.
Across all ecosystems, preserving the old interface alongside a new one is a very common approach to mitigate an immediate impact of a change on downstream users. While specifics depend on the language and tools, documenting methods as deprecated and providing default implementations for new extension points or parameters are common strategies to avoid breaking downstream implementations. In all these cases, the package developer invests additional effort now to preserve backward compatibility and accepts technical debt in the package for maintaining the extra code until it is eventually removed, in exchange for preventing an immediate impact of the change. The developer may at some later time clean up the code, affecting downstream users that have not updated in the meantime.
Similarly, many interviewees told us about various techniques to perform changes without breaking binary compatibility. They prevent rework costs for existing users by accepting more complicated implementations and harder maintenance in the changed package, while possibly also creating costs for new downstream users who have to deal with more complicated mechanisms.
Several developers reported strategies to maintain multiple parallel releases, such that downstream developers can delay updating or skip releases with larger breaking changes, while still incorporating minor nonbreaking changes (e.g., bug fixes) from a separate release. Node.js/npm offers specific mechanisms to support parallel releases with different version numbers; it is a common practice to provide security patches also for older releases. In contrast, CRAN only supports sequential version numbering, causing some developers to fork their own packages (e.g., `reshape' to `reshape2'). In each case, developers invest significant additional effort in maintaining old releases to reduce the (immediate) impact on downstream users.
A variant of this strategy is to maintain separate interfaces for different user groups with different stability commitments within the same package (see the façade pattern). For example, one Eclipse interviewee provided in parallel both a detailed and frequently changing API for expert users and a simpler and stable API that insulated less sophisticated users from most changes. Similarly, an R interviewee has split packages into smaller packages, with the intention that each user could depend only on parts relevant to them and would be exposed to less change. In both cases, the developer accepts the higher design and maintenance costs of multiple APIs for reduced impact on specific groups of users with distinct needs.
Some individual developers and some communities are considerate of downstream users when planning when to release changes. Some developers report deliberately delaying changes to batch multiple changes together. For example, an R interviewee keeps versions of his package with a quickly-changing API in a separate repository and updates CRAN less frequently when he wants to release a version to a broader audience. While in R/CRAN and Node.js/npm packages are usually released independently, large parts of the Eclipse community coordinate around synchronized yearly releases (a strategy also common in other package systems as Debian and Bioconductor). Delaying releases may incur coordination overhead and opportunity costs in slowing down development for the changer, but reduces the frequency (though not necessarily the severity) with which downstream users are exposed to changes and gives downstream users a planning horizon.
Finally, developers use various forms of communication with downstream users to reduce the impact of a breaking change. Several interviewees made early announcements to create awareness and receive feedback. For example, one interviewee explained that
two weeks or a month before the actual release, I do sort of a pre-release announcement on Twitter [and] tell people to use the README. Since our interview he has written a script to email all downstream maintainers before a release.
Another reason for communicating with downstream users was to help them deal with the aftermath of change. In the simplest case, a developer could invest effort in documenting how to upgrade. Several interviewees, were aware of their users and reached out to them individually; for example on interviewee contacted users using an old API to help them migrate, and another had most users present on-site and could therefore help them migrate their code. Another went so far as to create individual patches for downstream packages to get them to adopt a new interface and move away from an old deprecated one. In all cases, package maintainers invest effort to reduce costs for downstream users.
Just as package developers have some flexibility in planning changes that may affect downstream users, developers have flexibilities regarding
whether, when, and how to react to upstream change, again influenced by values, policies, and technologies. Having to monitor and react to upstream change can be a significant burden on developers (e.g., mismatch between schedules has been shown to be a barrier to collaboration). The urgency of reacting to change can depend significantly on the development context and platform mechanisms.
When discussing how frequently they react to upstream change, our interviewees described a spectrum ranging from never updating to closely monitoring all changes in upstream packages. Some interviewees explicitly ignored certain upstream changes; others upgraded dependencies only during their own releases or during deliberate house-cleaning sweeps. Even when the platform does not require updates, developers often prefer to update their dependencies to incorporate new fixes and features or to avoid accumulating technical debt. But they avoid updating when updates require too much effort (e.g., by causing complicated conflicts) or cause too much disruption downstream.
When developers have to or want to react in a timely fashion to upstream changes, they need to monitor the upstream projects in some way. The platform itself, e.g., Node.js, R core, and the CRAN infrastructure, is often an additional source of changes that developers need to keep up with. In our interviews, we discovered many different strategies for monitoring, including technical and social strategies. Their strategies varied along with the urgency of their needs, from active monitoring of upstream activity, to general social awareness of upstream activities, to a purely reactive stance where developers wait for some kind of notifications.
Only a few interviewees reported actively monitoring upstream changes by regularly looking at all changes of their upstream dependencies. Only three said they used GitHub's notification feed with some regularity. Several interviewees indicated that such raw notification feeds, in their current form, are a significant burden with a low signal to noise ratio, as stated for example by
The quantity of notifications I get on GitHub already is to the point of overwhelming. So I don't even mostly read them unless I'm actually working on the project at that moment. He later told us that after our interview he tried scaling back to watching just the 3-5 projects he is actively working on. Only one interviewee did not feel overwhelmed, saying that occasional, casual skimming of GitHub feeds was useful way to get a casual overview of activity.
In several cases, developers monitored upstream changes not as outsiders following a stream of data, but as active participants in those projects, collaborating to influence them toward their own needs or providing direct contributions to those packages.
Others actively compiled and tested their project with development versions of upstream dependencies, emphasizing the importance of giving timely reactions:
if you report it within a week there's a better chance the developer might remember what they did [...] which provides a good chance that they can revert their change before they hit their milestone.
Many interviewees tried to maintain a broad awareness of change through various social means. The most frequently mentioned mechanism, especially in the Node.js community, was Twitter. For example, one interviewee commented
the people who write the actual software are fairly well connected on Twitter, [...] like water cooler type of thing. So we tend to know what's going on elsewhere. In each ecosystem, several interviewees mentioned the importance of face-to-face interactions at conferences for awareness about important changes in the ecosystem. Other mentioned social mechanisms to learn about change were personal networks, blogs, and curated mailing lists. Though these mechanisms are rarely specific to individual packages, several developers mentioned them as their main monitoring strategies.
Although our research questions led us to probe interviewees about the aforementioned active and social monitoring practices, in fact most interviewed developers adopted a reactive strategy for most of their dependencies. They wait to hear about problems from others (in advance, or after things had broken): upstream developers contacting them about breaking changes, failing tests after dependency updates, or platform maintainers warning of changes that would affect them. There are also tools that enable this reactive stance, by creating targeted notifications on certain kinds of changes. The specific tools differ among the platforms and support different practices or policies. Policies and common practices (e.g., testing practices) in the platform strongly affect the reliability of a reactive strategy and corresponding tools.
Some Eclipse and Node.js/npm developers use continuous integration to detect compile-time issues caused by breaking changes in upstream packages early. For Node.js/npm, some developers use the tools gemnasium and greenkeeper to get notifications about new releases of upstream packages. Gemnasium alerts developers of package releases that fix known vulnerabilities, whereas greenkeeper submits pull requests to automate a continuous integration run against the new release. In either case, developers can react to notifications by email or pull requests.
CRAN is interesting because, by asking upstream developers to notify their users, it encourages downstream developers across the ecosystem to take a reactive stance (in contrast to Eclipse and Node.js/npm, where individual downstream developers need to employ optional monitoring tools). Some interviewees defended the practice of waiting to be told about breaking changes as a principled attention-preserving choice, consistent with ecosystem norms; others, however, were apologetic about being reactive:
I guess I'll sound crass about this and say it. For things like that I would wait to hear from CRAN when something broke. Because I don't think I can keep up with all of it. CRAN enforces this policy with manual and automated checking on each package update, running the package's tests and the test of all downstream packages in the repository, as well as some static checks. The CRAN team may then warn an affected downstream developer of an upcoming change by email.
Many developers have developed strategies to reduce their exposure to change from upstream modules and, thus, reduce their monitoring and rework efforts. The degree to which developers adopt such mitigation strategies again depends on the technology, policies, and values, as we will discuss.
Most of the CRAN and Eclipse interviewees that we asked, felt that it was better to have fewer dependencies. Reasons for limiting dependencies included limiting one's exposure to upstream changes and not burdening one's users with a lot of modules to install and potential version conflicts ("dependency hell"). One Eclipse interviewee represents a common view:
I only depend on things that are really worthwhile. Because basically everything that you depend on is going to give you pain every so often. And that's inevitable. Apart from removing no longer needed dependencies (tooling provided in Eclipse), some developers took more aggressive actions to avoid dependencies, including copying or recreating the functionality of another package.
In contrast, due to Node.js/npm's ability to use old versions and Eclipse's stability, some developers specifically said that they didn't see dependencies as a burden.
When limiting themselves to appropriate dependencies, interviewees mentioned a variety of different signals they looked for, such as trust of the developers, activity levels in the project, size and identiy of the user base, the project's history, and the quality of project artifacts.
Interestingly, there was almost no mention of traditional encapsulation strategies to isolate the impact of changes to upstream modules, contra to our expectations and typical software-engineering teaching. Only one Node.js developer mentioned developing an abstraction layer between his package and an upstream dependency.
Perhaps the coordination cost of following upstream packages, learning about changes, and understanding their impact is great enough that it dwarfs the code editing cost needed to make many changes in tightly coupled use of an upstream API.