Early permissioned data proposal draft feedback

Working Groups WG Private Data
1

This is a forum thread for giving feedback on the early permissioned data protocol proposal that I recently posted.

11 Likes
Permissioned data proposal discussion
What is an atmospheric group/community?
2

From the PR:

0016 Permissioned Data

This is a draft proposal, not the final specification. Details, terminology, and behaviors are all likely to change.

Introduction

The [AT Protocol][ATPROTO] is a protocol for public broadcast data. Users publish records into a repository on their PDS, and applications crawl those repositories to build views. Authority rests in the DID that publishes a record, and records are signed, redistributable, and universally addressable.

This document specifies a permissioned data protocol for data that is not public, data with an access perimeter. It runs alongside the public protocol and serves modalities such as:

  • Personal data: bookmarks, mutes, drafts
  • Gated content: paid newsletters, subscriber-only posts
  • Socially shared: private posts, stories
  • Groups: private forums, communities, group chats

The permissioned protocol shares the abstract shape of public atproto. It retains identity-based authority, per-user repositories, lexicon-typed records, and the general flow of applications crawling PDSes to build views. However it is a distinct protocol rather than an extension of the public one. It has its own repository format, sync mechanism, addressing scheme, and resolution path. Public atproto is built for public broadcast (signed, archival, rebroadcastable) while the permissioned protocol is built for party-to-party transmission within an access boundary.

This protocol provides access control, not confidentiality. It is not end-to-end encrypted. Services (both PDSes and authorized applications) can read the data they handle, which is required for server-side features such as search, indexing, notifications, aggregation, and moderation. E2EE is a separate concern that may be layered on top by an application and is out of scope in this proposal.

Relationship to public atproto

Public atproto Permissioned protocol
Unit of data Record in a repo Record in a permissioned repo
Repo scope One repo per user One permissioned repo per (user, space)
Record authority User DID User DID
URI authority User DID Space authority DID
Commit Merkle Search Tree root LtHash set-hash digest
Signature Rebroadcastable, archival Deniable on rebroadcast
Addressing at:// URI ats:// URI
Access Public Gated by space credential

Terminology

  • Space: an authorization and sync boundary for a set of permissioned records, identified by an (authority, type, skey) triple.
  • Permissioned repo: one user’s records within one space, with a cryptographic commit, hosted on that user’s PDS.
  • Repo host: a service that stores and serves users’ permissioned repos.
  • Space host: a service that answers for a space as a whole, issuing credentials, enumerating writers, and routing notifications.
  • Space authority: the DID at the root of a space, which resolves to the space host and the key material for issuing credentials.
  • Space credential: a token issued by the space authority that grants read access to a space.
  • Delegation token: a token issued by a user’s PDS that an application exchanges with a space authority for a space credential.
  • Client attestation: a token signed by an application’s own client authentication key, proving the application’s identity to a space authority. Required only when a space gates on app identity.
  • Syncer: an application that keeps its own copy of a space in sync by pulling from repo hosts.

A PDS fulfills both the roles of a repo host and a space host. However, these roles are discussed separately because they do not necessarily need to be filled by a PDS. A permissioned repo or a space may be hosted by any service that implements the required APIs.

Spaces

A space is an authorization and sync boundary representing a shared social context. A space may include many different types of records from many users. The space does not colocate records. Instead, each user stores their own records for a given space in a permissioned repo on their own repo host. A space is the union of these per-user repos across the network: an application presenting a space pulls each member’s repo from its host, assembles the view, and applies access control to requesting users.

Each space is identified by three values:

  1. authority: a DID, the root of authority for the space
  2. type: an NSID describing the modality of the space
  3. space key (skey): a string distinguishing spaces of the same type under the same authority

Reading or syncing a space requires a space credential signed by the declared signing key of the space authority. The space authority decides whether to issue one based on the requesting user and client application. The protocol does not define how that decision is made and carries no member list (see Access Control). Spaces scale from a single user’s personal data (e.g. bookmarks) to communities of millions of users.

Addressing

A permissioned record is addressed by an ats:// URI of six components:

ats://{spaceDid}/{spaceType}/{skey}/{authorDid}/{collection}/{rkey}
Component Type Description
spaceDid DID Space authority DID
spaceType NSID Space type
skey string Space key
authorDid DID DID of the record’s author
collection NSID Record collection
rkey string Record key

All six URI segments are necessary to identify a permissioned record. The first three components may be used to reference a space:

Space:  ats://{spaceDid}/{spaceType}/{skey}
Record: ats://{spaceDid}/{spaceType}/{skey}/{authorDid}/{collection}/{rkey}

Space authority

A space’s authority is the DID at the root of the space and the issuer of its credentials. It may be a user’s own DID as for personal data such as bookmarks or mutes. Or it may be a dedicated DID which lets a shared space transfer between users independently of any individual account.

The authority DID MUST expose two entries in its DID document:

  • a verification method with id #atproto_space: the public key used to verify the space’s credentials
  • a service entry with id #atproto_space_host: the endpoint of the space host

Both values MAY resolve to the same values as #atproto and #atproto_pds from the public data protocol.

Space type

A space’s type is an NSID that names its modality and resolves to a space declaration. It identifies the kind of data a space holds before any network resolution, much as a collection NSID does in public atproto. Because a type names a concrete modality, every space is some specific kind of space rather than a generic container.

The type is also the OAuth consent boundary. Access is granted to a user by type, e.g. “access to your AtmoBoards forums” (see OAuth scopes).

Snipped because too long, read the full PR

1 Like
3

Frankly beautiful, and I don’t say that often.

This covers all uses I have in mind myself so far quite nicely and, more importantly, appears to be a user-safe design that doesn’t leak attestations.

The simplespace feature requirement for PDSs is an important access-to-technology measure, so I’m very glad you included that here. (Its listMembers endpoint is private to the authority of the space, right?)
I think keeping that simple and not including authority ownership in the API there is probably fine, or at least I think client-only apps can still manage a form of transferable space authority ownership by storing credentials privately under this model. (Though, I suspect the owned-authority OAuth login can’t be fully automated in a normal browser with just this. Hm… But I think that’s a separate problem.)

In this line:

Space permissions can also be bundled, unlike with more user-friendly verbiage, into a permission set.

Is “unlike” a typo for “usually”?

2 Likes
4

Yes it is, thanks for flagging!

Yes as described, it is. Although if a space expresses the members as records in the space, then it will be viewable by anyone with read access to the space.

2 Likes
5

…which would be the only thing in that space – e.g. a space of type member records – is that correct?

1 Like
6

Awesome!

Since the proposal right now says terminology is still likely to change, I’d like to flag a potential naming issue early.

In everyday usage, “space” almost always refers to a place you’re in. However, the PD primitive is a permissioning boundary, a far more abstract concept. That conflict between a “place” and a “boundary” is, in my opinion, strong enough to count as a misnomer: it mismodels the concept for anyone meeting it for the first time.

I think that @zicklag.dev’s initial Arbiter design leaflet illustrates this well, especially considering that this seems to be where the group-permissioning layer is heading:

From now on, I’m going to use the terms “group”, “role”, and “space” somewhat interchangeably, based on how a specific space is meant to be used, even though they’re all just spaces.

If a role (a set of people holding a permission) is a “space” (colloquially, a place), then the word has lost its intuitive meaning; readers have to actively fight their intuition to follow the architecture.

I’m usually not one for nomenclature wars, but I do think that this one is particularly important: it’s a core concept that all of PD hangs from, and the cost of the mismodeling lands squarely on newcomers, who are a key part of the adoption boundary.

I wouldn’t be against a wholesale rename if that becomes consensus, but I think a fair compromise might be adopting “Permission Space” as the canonical term, with a capitalized “Space” as shorthand once context is set. This way, we could use the full “Permission Space” on first reference and anywhere the ambiguity is a real problem, and use the shorthand elsewhere. Compound terms already supply the needed context and could stay as they are (“Space credential,” “Space host,” “Space authority,” “Space type,” etc.), but the standalone “Space” in prose that does the most damage could be disambiguated by using its full qualified form.

I know it’s a small thing against a draft this early, but I do think naming is load-bearing for legibility and adoption, especially for a primitive this central, so I wanted to get it on the record.

7 Likes
Terminology of permissioned data
Terminology of permissioned data
8

I started a sidebar topic to test our appetite for further terminology bikeshedding: Terminology of permissioned data

3 Likes
9

My main thoughts are that access control is actually naturally stateful and data driven. “Does this user have permission to do this?” is a stateful question asked at call time somewhere in the call stack and the permission itself is data.

Also, I also don’t think terminology concerns are necessarily bike shedding. Counterintuitive terms are pedagogically expensive, naming things well does matter, and it’s harder to change the names of things when the protocol is already in place.

2 Likes
10

Another thought - the space owner’s PDS is a single point of failure for the whole space. That PDS must stay online to keep delivering tokens, or else eventually they expire and everyone loses access. I would like some distributed or redundant authority.

2 Likes
11

I think that can be done with any of the methods used to make distributed / redundant web services.

You can have the DNS for your space host contain multiple IP addresses, use a reverse proxy that routes to multiple backends, you can use a distributed database for membership lists if you need it, etc.

I don’t think that needs to be built-in to the protocol, because that is all stuff available to us through DNS, HTTP, etc.

For some projects it’s stateful and based on a member list that is stored on the space host, but in other projects it will be “stateless” and based on the result of a call to a 3rd party API like GitHub ( imagine allowing access to the space to any contributor to a GitHub repo with keytrace link to their ATProto account ).

The proposal allows different space hosts to implement the credential check however they need to.

In these “stateless” use-cases it’s actually very easy to make it highly available / redundant because the service itself can be made out of as many servers as you want running around the world and the rest of the reliability is built on top of GitHub’s API.

3 Likes
12

Fair point about a single DID having redundancy behind it.

Also, fair point that sometimes the permission will come from a 3rd party API. I still think a layer between this protocol and applications would be useful, but I suppose lexicons are already that layer, in principle. I think I am then interested in lexicons working the same for both public and permissioned data, which they should because it would be very unlike atproto if they didn’t.

1 Like
13

Yes Lexicons work the same for public & permissioned data.

I actually expect 3 layers (at least for the usecase of communities/groups): the protocol, a cross-app community governance/structure standard, and then application Lexicons.

We’re discussing what an atmospheric community is over here: What is an atmospheric group/community?

And I shared some early thoughts around some data modeling questions in this blog post: Modeling communities on permissioned data - Daniel's Leaflets

5 Likes
14

I took some high-level notes while giving PR review feedback, i’m going to dump them here publicly.

  • There has been some mixed feelings about the design process (eg, bsky team driving a proposal), and some projects are already building/operating alternative architectures. But as a general vibe there seems to be growing ecosystem consensus around this branch of development.
  • Nothing is set in stone until we have multiple interoperating implementations and a real application built around it. this includes both small details and the overall architecture.
  • App services having possession of multiple space credentials from multiple accounts for the same space does feel weird, and will lead to weird/arbitrary implementation decisions (as was raised early on by divy). I don’t think it is a blocking issue, but I hope we can bring structure to this. Eg, provide simple implementation patterns/guidelines.
  • This design enables interoperation, and retains some of the principles from public data. But I think the centralizing forces and anti-competitive behaviors are different for spaces (vs public “shared heap”), and we don’t have as deep an analysis or narrative around that yet.
  • Could use a careful security and privacy review from somebody with fresh eyes.
  • The moderation and anti-abuse story is not very fleshed out yet.
    • may want generic space blob pre-scanning at the PDS?
    • relationship between spaces, app services, and labelers needs to be worked out
    • human moderator access to reported content needs to be worked out
    • rate limits and resource quotas will be needed

Here is a sketch set of “design goals” and “architecture properties” that have come up through the process:

  • scales to millions of readers (eg, for newsletter use case)
  • personal data (eg, preferences) works with multiple devices, client apps, and app services
  • the authority for space records is rooted in the account’s DID (keys in DID document), not the account’s PDS hostname. eg, hijacking the PDS DNS should not allow an attacker to publish fake data. this is the same as public data repos.
  • newly authorized services can “backfill” an entire space (all space records from all members). this is critical for competition and credible exit from service providers.
  • support for multiple concurrent client apps, and services (eg, moderation services), in the same space. but at the same time, may want the space authority to be able to limit or exclude client apps and services.
  • ability to migrate spaces between hosts and services
  • an account’s individual permissioned data can be bulk exported, and migrates between PDS hosts the same as public data
  • services can verify the authenticity of data they fetch from PDS hosts; but leaked data is refutable (not strongly authenticated at rest). end users would mostly trust their client app software and app services to have verified authenticity, but in most cases should be able to re-verify individual records by re-fetching (eg, using goat, unless the space has locked down client app list)
7 Likes
15

What happens when a user leaves a space? What’s the responsibility of the app view then?

16

SO happy to see this come to life and really appreciate the openness and series of Leaflet posts about this! :clap::folded_hands:

Background: I build Sifa (a professional reputation product on atproto) and I have several feature ideas that could very much use permissioned data + we’re based in :european_union:*, so wanted to flag a few gaps from that angle.*

Deliberately not storing things like contact details today, because the two options we had were “public in the PDS” or “we become the data controller,” and neither one fit for now (especially with PD in our future).

My points partly touch on some EU-relevant legal/GDPR stuff and IANAL so take that into account :slight_smile:

1. Reading permissioned data means every aggregator holds and processes the data itself.

“Access control, not confidentiality” answers who may read, not what a reader becomes once it reads. The model has syncers who keep their own copy, and the only retention bound is the best-effort whole-space delete. So an app that pulls personal data through a space credential to render a view is processing it on its own account, and the spec offers no lower-exposure alternative: read_self is own-repo-only, and there’s no “fetch on demand, don’t persist” reader mode.

There’s also no easy way for the person sharing to see who can read what they shared: @dholms.xyz confirmed above that listMembers is private to the space authority (with the caveat that a space could expose members as records), so by default a member can’t enumerate who else can read the space. That last part isn’t only a UX gap. Under GDPR it’s a specific obligation: the right of access (Art. 15) lets a person ask for the recipients or categories of recipient of their data, and a reader holding a copy can’t answer that if the protocol won’t surface who the readers are.

Not asking the protocol to encode GDPR, but flagging that the design currently makes the lowest exposure path the hardest one to build, which might be worth fixing before shipped. @bnewbold.net asked for a privacy review with fresh eyes and this is one concrete thing it should probably answer?

2. Natural expiry isn’t erasure

The spec’s only mechanism for access ending is expiry, not revocation: space credentials are short-lived (default 2 hours), and “when [an app] loses all OAuth sessions for a space, it can no longer renew the credential and loses access” (spec, PR #94). This answers access control, but not erasure.

When someone deletes a contact detail they shared, the right to erasure (GDPR Art. 17) needs the copies already synced to actually be removed, not just to stop being renewed. And GDPR Art. 19 expects that erasure to be communicated to every recipient the data reached, which is basically the per-record/per-member purge signal the protocol doesn’t have yet.

Today the only affirmative purge signal is notifySpaceDeleted, which is whole-space. There’s no per-record or per-member “evict this”, and delete delivery over the oplog is best-effort with no confirmation that every cached copy complied (this is also the unanswered half of @j12t.org’s question about what happens when a user leaves).

The deletion side of the protocol is still upcoming (Diary 6 ends by noting the next post covers the sync protocol), so this is more a flag for that design: for personal data, it’d be great to have an affirmative, per-record or per-member purge with delivery confirmation, not only expiry. :folded_hands:

3. Trust and anti-abuse signals go dark off the public firehose

Reputation and sybil-detection systems (Sifa’s included), run over the public firehose, because it broadcasts every record with no per-space credential. 1-1 Endorsements, connections, and meetings are graph edges that feed that computation.

Move them into spaces and a network-wide labeler can’t see them, since it would need read credentials for every space it wanted to analyze, which doesn’t scale and defeats the point of permissioning. Some anti-abuse designs then undercut themselves, because a bot-cluster detector can only devalue hollow attestations it can see.

The output side is (yet) unspecified too: a label about permissioned content could be public, which can reveal that the content exists and was flagged, or permissioned, which confines it to that one space so it can’t protect the rest of the network.

@bnewbold.net noted the moderation and labeler story needs definition; for any reputation product built on permissioned signals, the date being available is load-bearing.

If any of these are worth it: happy to write them up on github :slight_smile:

6 Likes