Overview
Gerrit accounts are stored in NoteDb.
The account data consists of a sequence number (account ID), account properties (full name, display name, preferred email, registration date, status, inactive flag), preferences (general, diff and edit preferences), project watches, SSH keys, external IDs, starred changes and reviewed flags.
Most account data is stored in a special All-Users
repository, which has one branch per user. Within the user branch there
are Git config files for the
account properties, the account preferences and the
project watches. In addition there is an
authorized_keys
file for the SSH keys that follows
the standard OpenSSH file format.
The account data in the user branch is versioned and the Git history of this branch serves as an audit log.
The external IDs are stored as Git Notes inside the
All-Users
repository in the refs/meta/external-ids
notes branch.
Storing all external IDs in a notes branch ensures that each external
ID is only used once.
The starred changes are represented as
independent refs in the All-Users
repository. They are not stored in
the user branch, since this data doesn’t need versioning.
The reviewed flags are not stored in Git, but are persisted in a database table. This is because there is a high volume of reviewed flags and storing them in Git would be inefficient.
Since accessing the account data in Git is not fast enough for account queries, e.g. when suggesting reviewers, Gerrit has a secondary index for accounts.
All-Users
repository
The All-Users
repository is a special repository that only contains
user-specific information. It contains one branch per user. The user
branch is formatted as refs/users/CD/ABCD
, where CD/ABCD
is the
sharded account ID, e.g. the
user branch for account 1000856
is refs/users/56/1000856
. The
account IDs in the user refs are sharded so that there is a good
distribution of the Git data in the storage system.
A user branch must exist for each account, as it represents the account. The files in the user branch are all optional. This means having a user branch with a tree that is completely empty is also a valid account definition.
Updates to the user branch are done through the Gerrit REST API, but users can also manually fetch their user branch and push changes back to Gerrit. On push the user data is evaluated and invalid user data is rejected.
To hide the implementation detail of the sharded account ID in the ref
name Gerrit offers a magic refs/users/self
ref that is automatically
resolved to the user branch of the calling user. The user can then use
this ref to fetch from and push to the own user branch. E.g. if user
1000856
pushes to refs/users/self
, the branch
refs/users/56/1000856
is updated. In Gerrit self
is an established
term to refer to the calling user (e.g. in change queries). This is why
the magic ref for the own user branch is called refs/users/self
.
A user branch should only be readable and writeable by the user to whom
the account belongs. To assign permissions on the user branches the
normal branch permission system is used. In the permission system the
user branches are specified as refs/users/${shardeduserid}
. The
${shardeduserid}
variable is resolved to the sharded account ID. This
variable is used to assign default access rights on all user branches
that apply only to the owning user. The following permissions are set
by default when a Gerrit site is newly installed or upgraded to a
version which supports user branches:
[access "refs/users/${shardeduserid}"] exclusiveGroupPermissions = read push submit read = group Registered Users push = group Registered Users label-Code-Review = -2..+2 group Registered Users submit = group Registered Users
The user branch contains several files with account data which are described below.
In addition to the user branches the All-Users
repository also
contains a branch for the external IDs and special
refs for the starred changes.
Also the next available value of the account
sequence is stored in the All-Users
repository.
Account Index
There are several situations in which Gerrit needs to query accounts, e.g.:
-
For sending email notifications to project watchers.
-
For reviewer suggestions.
Accessing the account data in Git is not fast enough for account queries, since it requires accessing all user branches and parsing all files in each of them. To overcome this Gerrit has a secondary index for accounts. The account index is based on Lucene.
Via the Query Account REST endpoint generic account queries are supported.
Accounts are automatically reindexed on any update. The Index Account REST endpoint allows to reindex an account manually. In addition the reindex program can be used to reindex all accounts offline.
Account Data in User Branch
A user branch contains several Git config files with the account data:
-
account.config
:Stores the account properties.
-
preferences.config
:Stores the user preferences of the account.
-
watch.config
:Stores the project watches of the account.
In addition it contains an authorized_keys file with the SSH keys of the account.
Account Properties
The account properties are stored in the user branch in the
account.config
file:
[account] fullName = John Doe displayName = John preferredEmail = john.doe@example.com status = OOO active = false
For active accounts the active
parameter can be omitted.
The registration date is not contained in the account.config
file but
is derived from the timestamp of the first commit on the user branch.
When users update their account properties by pushing to the user branch, it is verified that the preferred email exists in the external IDs.
Users are not allowed to flip the active value themselves; only administrators and users with the Modify Account global capability are allowed to change it.
Since all data in the account.config
file is optional the
account.config
file may be absent from some user branches.
Preferences
The account properties are stored in the user branch in the
preferences.config
file. There are separate sections for
general,
diff and edit preferences:
[diff] hideTopMenu = true [edit] lineLength = 80
The parameter names match the names that are used in the preferences REST API:
If the value for a preference is the same as the default value for this
preference, it can be omitted in the preferences.config
file.
Defaults for preferences that apply for all accounts can be configured
in the refs/users/default
branch in the All-Users
repository.
Project Watches
Users can configure watches on projects to receive email notifications for changes of that project.
A watch configuration consists of the project name and an optional filter query. If a filter query is specified, email notifications will be sent only for changes of that project that match this query.
In addition, each watch configuration can contain a list of notification types that determine for which events email notifications should be sent. E.g. a user can configure that email notifications should only be sent if a new patch set is uploaded and when the change gets submitted, but not on other events.
Project watches are stored in a watch.config
file in the user branch:
[project "foo"] notify = * [ALL_COMMENTS] notify = branch:master [ALL_COMMENTS, NEW_PATCHSETS] notify = branch:master owner:self [SUBMITTED_CHANGES]
The watch.config
file has one project section for all project watches
of a project. The project name is used as subsection name and the
filters with the notification types, that decide for which events email
notifications should be sent, are represented as notify
values in the
subsection. A notify
value is formatted as
"<filter> [<comma-separated-list-of-notification-types>]". The
supported notification types are described in the
Email Notifications documentation.
For a change event, a notification will be sent if any notify
value
of the corresponding project has both a filter that matches the change
and a notification type that matches the event.
In order to send email notifications on change events, Gerrit needs to find all accounts that watch the corresponding project. To make this lookup fast the secondary account index is used. The account index contains a repeated field that stores the projects that are being watched by an account. After the accounts that watch the project have been retrieved from the index, the complete watch configuration is available from the account cache and Gerrit can check if any watch matches the change and the event.
SSH Keys
SSH keys are stored in the user branch in an authorized_keys
file,
which is the
standard OpenSSH file format for storing SSH keys:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCgug5VyMXQGnem2H1KVC4/HcRcD4zzBqSuJBRWVonSSoz3RoAZ7bWXCVVGwchtXwUURD689wFYdiPecOrWOUgeeyRq754YWRhU+W28vf8IZixgjCmiBhaL2gt3wff6pP+NXJpTSA4aeWE5DfNK5tZlxlSxqkKOS8JRSUeNQov5Tw== john.doe@example.com # DELETED # INVALID ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDm5yP7FmEoqzQRDyskX+9+N0q9GrvZeh5RG52EUpE4ms/Ujm3ewV1LoGzc/lYKJAIbdcZQNJ9+06EfWZaIRA3oOwAPe1eCnX+aLr8E6Tw2gDMQOGc5e9HfyXpC2pDvzauoZNYqLALOG3y/1xjo7IH8GYRS2B7zO/Mf9DdCcCKSfw== john.doe@example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCaS7RHEcZ/zjl9hkWkqnm29RNr2OQ/TZ5jk2qBVMH3BgzPsTsEs+7ag9tfD8OCj+vOcwm626mQBZoR2e3niHa/9gnHBHFtOrGfzKbpRjTWtiOZbB9HF+rqMVD+Dawo/oicX/dDg7VAgOFSPothe6RMhbgWf84UcK5aQd5eP5y+tQ== john.doe@example.com
When the SSH API is used, Gerrit needs an efficient way to lookup SSH keys by username. Since the username can be easily resolved to an account ID (via the account cache), accessing the SSH keys in the user branch is fast.
To identify SSH keys in the REST API Gerrit uses
sequence numbers per account.
This is why the order of the keys in the authorized_keys
file is
used to determine the sequence numbers of the keys (the sequence
numbers start at 1).
To keep the sequence numbers intact when a key is deleted, a '# DELETED' line is inserted at the position where the key was deleted.
Invalid keys are marked with the prefix '# INVALID'.
External IDs
External IDs are used to link identities, such as the username and email addresses, and external identies such as an LDAP account or an OAUTH identity, to an account in Gerrit.
External IDs are stored as Git Notes in the All-Users
repository. The
name of the notes branch is refs/meta/external-ids
.
As note key the SHA-1 of the external ID key is used, for example the key
for the external ID username:jdoe
is e0b751ae90ef039f320e097d7d212f490e933706
.
This ensures that an external ID is used only once (e.g. an external ID can
never be assigned to multiple accounts at a point in time).
By default, the SHA-1 sum is computed preserving the case of the external ID. If
auth.userNameCaseInsensitive` is set to true
, the SHA-1 sum of external IDs
in the gerrit:
and username:
schemes are computed from the all lowercase
external ID. This enables case insensitive username handling. The case of the
external ID is however preserved by using the original capitalization in the
note content.
The following commands show how to find the SHA-1 of an external ID:
$ echo -n 'gerrit:jdoe' | shasum 7c2a55657d911109dbc930836e7a770fb946e8ef - $ echo -n 'username:jdoe' | shasum e0b751ae90ef039f320e097d7d212f490e933706 -
Important
|
If the external ID key is changed manually you must adapt the note key to the new SHA-1, otherwise the external ID becomes inconsistent and is ignored by Gerrit. |
The note content is a Git config file:
[externalId "username:jdoe"] accountId = 1003407 email = jdoe@example.com password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7
Once SHA-1 of an external ID is known the following command can be used to show the content of the note:
$ echo -n 'gerrit:jdoe' | shasum 7c2a55657d911109dbc930836e7a770fb946e8ef - $ git show refs/meta/external-ids:7c/2a55657d911109dbc930836e7a770fb946e8ef [externalId "username:jdoe"] accountId = 1003407 email = jdoe@example.com password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7
The config file has one externalId
section. The external ID key, which
consists of scheme and ID in the format '<scheme>:<id>', is used as
subsection name.
The accountId
field is mandatory. The email
and password
fields
are optional.
Note that git will automatically nest these notes at varying levels. If refs/meta/external-ids:7c/2a55657d911109dbc930836e7a770fb946e8ef is not found then check refs/meta/external-ids:7c/2a/55657d911109dbc930836e7a770fb946e8ef and so on.
The external IDs are maintained by Gerrit. This means users are not
allowed to manually edit their external IDs. Only users with the
Access Database
global capability can push updates to the refs/meta/external-ids
branch. However Gerrit rejects pushes if:
-
any external ID config file cannot be parsed
-
if a note key does not match the SHA of the external ID key in the note content
-
external IDs for non-existing accounts are contained
-
invalid emails are contained
-
any email is not unique (the same email is assigned to multiple accounts)
-
hashed passwords of external IDs with scheme
username
cannot be decoded
Users can edit some external IDs via the user settings page or the REST API. Note that email addresses cannot be deleted if they are associated with the user’s login credentials external ID, for example the email address associated with an OpenId or OAUTH external ID. If users wish to remove these email addresses from Gerrit they must first update the external authentication record in that system, log in to Gerrit, then Gerrit will update the external ID record with the new email address.
Transition from LDAP to Google OAuth
When authentication is changed from LDAP to Google Oauth gerrit will automatically
adjust the external IDs in the refs/meta/external-ids
branch. Gerrit will re-use
the same account ID that was used by the LDAP account. Transition to other OAuth
mechanisms will fail and require manual changes to the refs/meta/external-ids
branch.
The LDAP e-mail and Google OAuth e-mail must be the same.
Starred Changes
Starred changes allow users to mark changes as favorites and receive email notifications for them.
Each starred change is a tuple of an account ID, a change ID and a label.
To keep track of a change that is starred by an account, Gerrit creates
a refs/starred-changes/YY/XXXX/ZZZZZZZ
ref in the All-Users
repository, where YY/XXXX
is the sharded numeric change ID and
ZZZZZZZ
is the account ID.
A starred-changes ref points to a blob that contains the list of labels that the account set on the change. The label list is stored as UTF-8 text with one label per line.
Since JGit has explicit optimizations for looking up refs by prefix when the prefix ends with '/', this ref format is optimized to find starred changes by change ID. Finding starred changes by change ID is e.g. needed when a change is updated so that all users that have the star on the change can be notified by email.
Gerrit also needs an efficient way to find all changes that were
starred by an account, e.g. to provide results for the
is:starred query operator. With the
ref format as described above the lookup of starred changes by account
ID is expensive, as this requires a scan of the full
refs/starred-changes/*
namespace. To overcome this the users that
have starred a change are stored in the change index together with the
star labels.
Reviewed Flags
When reviewing a patch set in the Gerrit UI, the reviewer can mark files in the patch set as reviewed. These markers are called ‘Reviewed Flags’ and are private to the user. A reviewed flag is a tuple of patch set ID, file and account ID.
Each user can have many thousands of reviewed flags and over time the number can grow without bounds.
The high amount of reviewed flags makes a storage in Git unsuitable because each update requires opening the repository and committing a change, which is a high overhead for flipping a bit. Therefore the reviewed flags are stored in a database table. By default they are stored in a local H2 database, but there is an extension point that allows to plug in alternate implementations for storing the reviewed flags. To replace the storage for reviewed flags a plugin needs to implement the AccountPatchReviewStore interface. E.g. to support a cluster setup with multiple primary servers handling write operations where reviewed flags should be replicated between the primary nodes one could implement a store for the reviewed flags that is based on MySQL with replication.
Account Sequence
The next available account sequence number is stored as UTF-8 text in a
blob pointed to by the refs/sequences/accounts
ref in the All-Users
repository.
Multiple processes share the same sequence by incrementing the counter
using normal git ref updates. To amortize the cost of these ref
updates, processes increment the counter by a larger number and hand
out numbers from that range in memory until they run out. The size of
the account ID batch that each process retrieves at once is controlled
by the
notedb.accounts.sequenceBatchSize parameter in the gerrit.config
file.
Replication
To replicate account data the following branches from the All-Users
repository must be replicated:
-
refs/users/*
(user branches) -
refs/meta/external-ids
(external IDs) -
refs/starred-changes/*
(star labels) -
refs/sequences/accounts
(account sequence numbers, not needed for Gerrit replicas)
Part of Gerrit Code Review