ef

Updates

1. 18F’s Melody Kramer interviewed me about this project and I went deeper into some of my takeaways.
2. Open data strategist Rayna Stamboliyska translated parts of the blog into French for her DataColada newsletter.
3. The European Commission included this research in its 2016 annual report of the “most important findings, trends and technology shifts in open source and public administrations.”
4. The Free Software Foundation Europe referred to this post in its comments on the October 2017 Tallinn Declaration for e-government proposed by the Estonian presidency of the Council of the EU.

Governments have been flocking to GitHub.

Their reasons are plenty: the promise of “private sector” tools, a conviction that publicly-funded code should be public, the company’s evangelism (and stickers), etc. Whatever the case, GitHub now hosts at least 600 government organizations, with over 9,000 public repositories between them.

I had a notion of the global ecosystem this activity has sprouted—the players and their interactions—but wanted to back it up with data.

So, using GitHub’s API, I compiled a database of government GitHub organizations, their repositories, members, and contributors and dove in.

Contents

• The Government GitHub Network
• Appendix: Tables, Graphs, and Stats
  • Organizations
  • Repositories
  • Users

Summary

Overall, reuse within the government GitHub “ecosystem” is uneven and limited.

Nearly all popular repositories (inside and outside of government) were created by US and UK national organizations. The bulk are standards or frameworks. Modular products, like data.gov.uk’s CKAN extensions, also seem relatively reusable.

Collaborative work and reuse is most concentrated within the large US and UK national-level networks. This may point to the importance of scale, “real world” interactions (e.g. talks, meet-ups, employees switching between organizations), and the alignment of policy priorities, timelines, licensing, and tech stacks.

14% of repositories have no further activity after being posted to GitHub. 46% remain under development a year after they were created.

I didn’t find a license file for half of the repositories. At least 13% use the MIT license. At least 8% use some version of the GPL. License choice varies geographically.

Government GitHub organizations are bringing some new users to the platform along with them. But 45% of the users predate the government organizations they contribute to.

Estonia has the most government repositories per capita at 72.8 per million residents (hover over and click to zoom in on the map up top).

Notes and Caveats

• The code to generate the database is on GitHub.
• The list of government GitHub organizations is certainly incomplete—add more if you know any!
• Unless otherwise specified, ‘repositories’ refers to repositories that are not themselves forks.
• All repository, membership, and contribution statistics include only public information. I assume most repositories are kept public. Organizational membership, however, is private by default.
• The GitHub API only checks for license files at the root of the repository. Some may be embedded in the README or placed in sub-folders.

The Government GitHub Network

Government development teams interact and influence one another through various channels. These include:

• sharing members or repository contributors;
• forking, starring, or cloning each others’ repositories (and possibly submitting pull requests)
• contributing issues;
• reading each others’ code, READMEs, and blogs; and
• talking to each other in person or on other platforms.

Because the GitHub API gives us access to membership, contribution, and forking relationships, I’ll focus on those.

Contribution Network

view full size

This graph shows 277 nodes (or organizations) connected to one another with 1270 edges. If two organizations share an edge, they have at least one contributor in common.

Overall, 941 unique users account for 2751 individual contributor connections between the organizations (see user statistics).

323 organizations are “loners,” with no contributors shared with other organizations. These don’t appear in the graph.

The thicker edges represent more contributors in common. Nodes are sized by the number of other nodes they’re connected to (their “degree”). If you view the full size graph, each node links to its GitHub site.

Two main clusters stick out. Up top in bright green are UK national organizations. On the right in purple are the US federal organizations.

The City of Philadelphia has the most prominent non-national organization. You might also notice the DC Government in the mix, as well the USGS/NOAA, Brazil, Canada, and Australia sub-networks.

It’s likely some of the connections aren’t real, but are the artifacts of cloned repositories. These may retain the original repository’s contribution history in addition to any new commits, but the API won’t mark them as forks.

This turns the contributor graph into a mish-mash of genuine collaboration of one organization’s members with another’s, non-members who contribute code to multiple organizations, and reuse.

Membership Network

We can do the same thing with shared members.

view full size

This graph of 96 nodes (organizations) is tied together using 148 edges. Behind these edges are 327 individual member connections from 137 unique users (see user statistics).

Again, we find the highly inter-linked US federal sub-network. There are also the smaller UK and Canada membership sub-networks.

GitHub makes membership private by default, so there are likely more member connections in reality. But, in general, it makes sense that this graph would be much sparser than the contribution graph.

Why is the US federal sub-network comparatively dense? Many of them have large memberships, so there are more potential connections to be made. Some (like 18F) have a policy requiring that staff make their membership public. A number operate as consultancies to other federal agencies. And, from what I’ve seen, many “techies” enter the US federal government through one of these agencies and then hop around.

Forking Network

view full size

This graph shows organization forking connections. Arrows connect the forked repository’s source → to its destination.

121 organizations have forked other organizations’ repositories 223 times (138 edges). For comparison, 1858 forks come from non-government organizations or users and 9032 repositories are not forks.

Most of the graph’s forks go unreciprocated; only 8 government organizations (mainly US federal) have forked one another’s repositories (↔).

data.gov.uk is in an unusual position, being disconnected from the rest of the UK, while the source for organizations in other countries (Romania, Estonia, Paraguay, Canada, and US).

Open data projects, like data.gov.uk’s CKAN contributions, seem poised for cross-border reuse. This may be because use cases are quite standardized and modular extensions address any differences.

Organization, Repository & User Statistics

I provide some additional views into government GitHub organization, repository, and user data below.

There are certainly many other questions to look into. Check out this repository to generate your own database (or reuse the one there).

Organizations

The list I used included 600 government GitHub organizations. You can see their geographic distribution on the map up top.

No. repositories

Of note, not only does UK's Government Digital Service make the top 10, but so does its GitHub organization for retired repositories! Neat appearances from the Norwegian Meterological Institute, the Gemeinsamer Bibliotheksverbund in Germany, and the National Library of Finland. 

This includes only repositories that are not themselves forks.

show more

No. contributors

show more

No. members

Member counts are likely quite a bit higher in reality. Because GitHub defaults to private membership, in my experience many users don't switch their preferences to be public. I bet that this isn't intentional in most cases. GitHub would do well to make this option more obvious when you first join an organization.

show more

No. times forked by others

show more

No. repositories that are forks

show more

Repositories

The data showed 11113 public repositories—2081 forked repositories and 9032 non-forked.

Licensing

Listed below are each license type, their frequency, the regions that most frequently use the license, and the percentage of each region's repositories with the license. 

I only include regions with at least 10 repositories and 2 organizations.

Note: The GitHub API only looks for a license file at the root of the repository, so licenses embedded in the README or stored in a subfolder are marked as having no license. Italy's not really that bad!

Age

Repository count by date created

Things have certainly picked up.

Oldest Repositories

show more

Of the 10 oldest, 4 have had commits in the past month.

Development Lifetime

Many government GitHub repositories have a fairly short development lifespan. 14% show no further development after after they were first pushed to GitHub. However, 46% were under development a year in, 18% two years in, and 6% three years in.

11 repositories have Git histories earlier than their initial push date. Seems like a lot of development happens locally and without version control, then the repository is dumped on GitHub.

No. contributors

show more

No. stars

Understandably, people are most interested in tools, standards, and frameworks. US Army's net-sec tool beating 18F was a surprise, though.

show more

No. forks (gov → gov)

show more

No. forks (gov → all)

show more

No. forks (non-gov → gov)

Government forks little from without and little from without. The government.github.com repository is likely from organizations adding themselves to the "official" list.

show more

Users

The data showed 7887 public contributors to government repositories (that are not forks) and 1512 public members of government organizations.

User join date vs. organization creation date

Are government GitHub organizations spurring new users to join the platform? We don't know the date users joined an organization, so we'll have to proxy.

The histograms below show the difference (in days) between 1) when a user joined GitHub and 2) the earliest creation date of all the government organizations to which they contribute or belong.

There's clearly a bump in the center. That spike shows users who joined GitHub at the same time as the government organization with which they're involved. Some of those users who came to the platform later may also have joined a government organization the same day as their arrival.

In each case, about half of users predate their organization. This says to me that public, social coding is generally new at an individual level—not just at an institutional one.

No. repositories contributed to

The top contributors (in terms of government repository count, at least) are all members of UK's Government Digital Service, 18F, or the Consumer Financial Protection Bureau.

A majority of users are one-off contributors (see percentiles).

show more

• 25th percentile: 1
• 50th percentile: 1
• 75th percentile: 3

No. organizations a member of

All but one (mattbostock of the UK) of the top ten members are part of the web of U.S. federal programs—the USDS-18F-PIF-CFPB-White House connection.

It's quite rare for a user to be a member of more than one organization.

show more

• 25th percentile: 1
• 50th percentile: 1
• 75th percentile: 1

Made it down this far?

Huzzah! You deserve a cookie.