1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174 |
- .\" Man page generated from reStructuredText.
- .
- .
- .nr rst2man-indent-level 0
- .
- .de1 rstReportMargin
- \\$1 \\n[an-margin]
- level \\n[rst2man-indent-level]
- level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
- -
- \\n[rst2man-indent0]
- \\n[rst2man-indent1]
- \\n[rst2man-indent2]
- ..
- .de1 INDENT
- .\" .rstReportMargin pre:
- . RS \\$1
- . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
- . nr rst2man-indent-level +1
- .\" .rstReportMargin post:
- ..
- .de UNINDENT
- . RE
- .\" indent \\n[an-margin]
- .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
- .nr rst2man-indent-level -1
- .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
- .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
- ..
- .TH "SYNCTHING-BEP" "7" "Feb 29, 2024" "v1.27.3" "Syncthing"
- .SH NAME
- syncthing-bep \- Block Exchange Protocol v1
- .SH INTRODUCTION AND DEFINITIONS
- .sp
- The Block Exchange Protocol (BEP) is used between two or more \fIdevices\fP thus
- forming a \fIcluster\fP\&. Each device has one or more \fIfolders\fP of files
- described by the \fIlocal model\fP, containing metadata and block hashes. The
- local model is sent to the other devices in the cluster. The union of all
- files in the local models, with files selected for highest change version,
- forms the \fIglobal model\fP\&. Each device strives to get its folders in sync
- with the global model by requesting missing or outdated blocks from the
- other devices in the cluster.
- .sp
- File data is described and transferred in units of \fIblocks\fP, each being from
- 128 KiB (131072 bytes) to 16 MiB in size, in steps of powers of two. The
- block size may vary between files but is constant in any given file, except
- for the last block which may be smaller.
- .sp
- The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
- “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this
- document are to be interpreted as described in \fI\%RFC 2119\fP <\fBhttps://datatracker.ietf.org/doc/html/rfc2119.html\fP>\&.
- .SH TRANSPORT AND AUTHENTICATION
- .sp
- BEP is deployed as the highest level in a protocol stack, with the lower
- level protocols providing encryption and authentication.
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
- | Block Exchange Protocol |
- |\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|
- | Encryption & Auth (TLS 1.2) |
- |\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|
- | Reliable Transport |
- |\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|
- v ... v
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .sp
- The encryption and authentication layer SHALL use TLS 1.2 or a higher
- revision. A strong cipher suite SHALL be used, with “strong cipher
- suite” being defined as being without known weaknesses and providing
- Perfect Forward Secrecy (PFS). Examples of strong cipher suites are
- given at the end of this document. This is not to be taken as an
- exhaustive list of allowed cipher suites but represents best practices
- at the time of writing.
- .sp
- The exact nature of the authentication is up to the application, however
- it SHALL be based on the TLS certificate presented at the start of the
- connection. Possibilities include certificates signed by a common
- trusted CA, preshared certificates, preshared certificate fingerprints
- or certificate pinning combined with some out of band first
- verification. The reference implementation uses preshared certificate
- fingerprints (SHA\-256) referred to as “Device IDs”.
- .sp
- There is no required order or synchronization among BEP messages except
- as noted per message type \- any message type may be sent at any time and
- the sender need not await a response to one message before sending
- another.
- .sp
- The underlying transport protocol MUST guarantee reliable packet delivery.
- .sp
- In this document, in diagrams and text, “bit 0” refers to the \fImost
- significant\fP bit of a word; “bit 15” is thus the least significant bit of a
- 16 bit word (int16) and “bit 31” is the least significant bit of a 32 bit
- word (int32). Non protocol buffer integers are always represented in network
- byte order (i.e., big endian) and are signed unless stated otherwise, but
- when describing message lengths negative values do not make sense and the
- most significant bit MUST be zero.
- .sp
- The protocol buffer schemas in this document are in \fBproto3\fP syntax. This
- means, among other things, that all fields are optional and will assume
- their default value when missing. This does not necessarily mean that a
- message is \fIvalid\fP with all fields empty \- for example, an index entry for a
- file that does not have a name is not useful and MAY be rejected by the
- implementation. However the folder label is for human consumption only so an
- empty label should be accepted \- the implementation will have to choose some
- way to represent the folder, perhaps by using the ID in it’s place or
- automatically generating a label.
- .SH PRE-AUTHENTICATION MESSAGES
- .sp
- AFTER establishing a connection, but BEFORE performing any authentication,
- devices MUST exchange Hello messages.
- .sp
- Hello messages are used to carry additional information about the peer,
- which might be of interest to the user even if the peer is not permitted to
- communicate due to failing authentication. Note that the certificate based
- authentication may be considered part of the TLS handshake that precedes the
- Hello message exchange, but even in the case that a connection is rejected a
- Hello message must be sent before the connection is terminated.
- .sp
- Hello messages MUST be prefixed with an int32 containing the magic number
- \fB0x2EA7D90B\fP, followed by an int16 representing the size of the message,
- followed by the contents of the Hello message itself.
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- 0 1
- 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- | Magic |
- | (32 bits) |
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- | Length |
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- / /
- \e Hello \e
- / /
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .sp
- The Hello message itself is in protocol buffer format with the following schema:
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Hello {
- string device_name = 1;
- string client_name = 2;
- string client_version = 3;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields (Hello message)
- .sp
- The \fBdevice_name\fP is a human readable (configured or auto detected) device
- name or host name, for the remote device.
- .sp
- The \fBclient_name\fP and \fBclient_version\fP identifies the implementation. The
- values SHOULD be simple strings identifying the implementation name, as a
- user would expect to see it, and the version string in the same manner. An
- example client name is “syncthing” and an example client version is “v0.7.2”.
- The client version field SHOULD follow the patterns laid out in the \fI\%Semantic
- Versioning\fP <\fBhttps://semver.org/\fP> standard.
- .sp
- Immediately after exchanging Hello messages, the connection MUST be dropped
- if the remote device does not pass authentication.
- .SH POST-AUTHENTICATION MESSAGES
- .sp
- Every message post authentication is made up of several parts:
- .INDENT 0.0
- .IP \(bu 2
- A header length word
- .IP \(bu 2
- A \fBHeader\fP
- .IP \(bu 2
- A message length word
- .IP \(bu 2
- A \fBMessage\fP
- .UNINDENT
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- 0 1
- 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- | Header Length |
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- / /
- \e Header \e
- / /
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- | Message Length |
- | (32 bits) |
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- / /
- \e Message \e
- / /
- +\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+\-+
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .sp
- The header length word is 16 bits. It indicates the length of the following
- \fBHeader\fP message. The Header is in protocol buffer format. The Header
- describes the type and compression status of the following message.
- .sp
- The message is preceded by the 32 bit message length word and is one of the
- concrete BEP messages described below, identified by the \fBtype\fP field of
- the Header.
- .sp
- As always, the length words are in network byte order (big endian).
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Header {
- MessageType type = 1;
- MessageCompression compression = 2;
- }
- enum MessageType {
- CLUSTER_CONFIG = 0;
- INDEX = 1;
- INDEX_UPDATE = 2;
- REQUEST = 3;
- RESPONSE = 4;
- DOWNLOAD_PROGRESS = 5;
- PING = 6;
- CLOSE = 7;
- }
- enum MessageCompression {
- NONE = 0;
- LZ4 = 1;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .sp
- When the \fBcompression\fP field is \fBNONE\fP, the message is directly in
- protocol buffer format.
- .sp
- When the compression field is \fBLZ4\fP, the message consists of a 32 bit
- integer describing the uncompressed message length followed by a single LZ4
- block. After decompressing the LZ4 block it should be interpreted as a
- protocol buffer message just as in the uncompressed case.
- .SH MESSAGE SUBTYPES
- .SS Cluster Config
- .sp
- This informational message provides information about the cluster
- configuration as it pertains to the current connection. A Cluster Config
- message MUST be the first post authentication message sent on a BEP
- connection. Additional Cluster Config messages MUST NOT be sent after the
- initial exchange.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message ClusterConfig {
- repeated Folder folders = 1;
- }
- message Folder {
- string id = 1;
- string label = 2;
- bool read_only = 3;
- bool ignore_permissions = 4;
- bool ignore_delete = 5;
- bool disable_temp_indexes = 6;
- bool paused = 7;
- repeated Device devices = 16;
- }
- message Device {
- bytes id = 1;
- string name = 2;
- repeated string addresses = 3;
- Compression compression = 4;
- string cert_name = 5;
- int64 max_sequence = 6;
- bool introducer = 7;
- uint64 index_id = 8;
- bool skip_introduction_removals = 9;
- bytes encryption_password_token = 10;
- }
- enum Compression {
- METADATA = 0;
- NEVER = 1;
- ALWAYS = 2;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields (Cluster Config Message)
- .sp
- The \fBfolders\fP field contains the list of folders that will be synchronized
- over the current connection.
- .SS Fields (Folder Message)
- .sp
- The \fBid\fP field contains the folder ID, which is the unique identifier of
- the folder.
- .sp
- The \fBlabel\fP field contains the folder label, the human readable name of
- the folder.
- .sp
- The \fBread_only\fP field is set for folders that the device will accept no
- updates from the network for.
- .sp
- The \fBignore_permissions\fP field is set for folders that the device will not
- accept or announce file permissions for.
- .sp
- The \fBignore_delete\fP field is set for folders that the device will ignore
- deletes for.
- .sp
- The \fBdisable_temp_indexes\fP field is set for folders that will not dispatch
- and do not wish to receive progress updates about partially downloaded files
- via Download Progress messages.
- .sp
- The \fBpaused\fP field is set for folders that are currently paused.
- .sp
- The \fBdevices\fP field is a list of devices participating in sharing this
- folder.
- .SS Fields (Device Message)
- .sp
- The device \fBid\fP field is a 32 byte number that uniquely identifies the
- device. For instance, the reference implementation uses the SHA\-256 of the
- device X.509 certificate.
- .sp
- The \fBname\fP field is a human readable name assigned to the described device
- by the sending device. It MAY be empty and it need not be unique.
- .sp
- The list of \fBaddresses\fP is that used by the sending device to connect to
- the described device.
- .sp
- The \fBcompression\fP field indicates the compression mode in use for this
- device and folder. The following values are valid:
- .INDENT 0.0
- .TP
- .B 0
- Compress metadata. This enables compression of metadata messages such as Index.
- .TP
- .B 1
- Compression disabled. No compression is used on any message.
- .TP
- .B 2
- Compress always. Metadata messages as well as Response messages are compressed.
- .UNINDENT
- .sp
- The \fBcert_name\fP field indicates the expected certificate name for this
- device. It is commonly blank, indicating to use the implementation default.
- .sp
- The \fBmax_sequence\fP field contains the highest sequence number of the files
- in the index. See \fI\%Delta Index Exchange\fP for the usage of this field.
- .sp
- The \fBintroducer\fP field is set for devices that are trusted as cluster
- introducers.
- .sp
- The \fBindex_id\fP field contains the unique identifier for the current set of
- index data. See \fI\%Delta Index Exchange\fP for the usage of this field.
- .sp
- The \fBskip_introduction_removals\fP field signifies if the remote device has
- opted to ignore introduction removals for the given device. This setting is
- copied across as we are being introduced to a new device.
- .sp
- The \fBencryption_password_token\fP field contains a token derived from the password, that is
- used to encrypt data sent to this device. If the device is the same as the
- device sending the message, it signifies that the device itself has encrypted
- data that was encrypted with the given token. It is empty or missing if there is
- no encryption. See \fI\%Untrusted Device Encryption\fP for details on the encryption scheme.
- .SS Index and Index Update
- .sp
- The Index and Index Update messages define the contents of the senders
- folder. An Index message represents the full contents of the folder and
- thus supersedes any previous index. An Index Update amends an existing
- index with new information, not affecting any entries not included in
- the message. An Index Update MAY NOT be sent unless preceded by an
- Index, unless a non\-zero Max Sequence has been announced for the
- given folder by the peer device.
- .sp
- The Index and Index Update messages are currently identical in format,
- although this is not guaranteed to be the case in the future.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Index {
- string folder = 1;
- repeated FileInfo files = 2;
- }
- message IndexUpdate {
- string folder = 1;
- repeated FileInfo files = 2;
- }
- message FileInfo {
- string name = 1;
- FileInfoType type = 2;
- int64 size = 3;
- uint32 permissions = 4;
- int64 modified_s = 5;
- int32 modified_ns = 11;
- uint64 modified_by = 12;
- bool deleted = 6;
- bool invalid = 7;
- bool no_permissions = 8;
- Vector version = 9;
- int64 sequence = 10;
- int32 block_size = 13;
- repeated BlockInfo Blocks = 16;
- string symlink_target = 17;
- }
- enum FileInfoType {
- FILE = 0;
- DIRECTORY = 1;
- SYMLINK_FILE = 2 [deprecated = true];
- SYMLINK_DIRECTORY = 3 [deprecated = true];
- SYMLINK = 4;
- }
- message BlockInfo {
- int64 offset = 1;
- int32 size = 2;
- bytes hash = 3;
- uint32 weak_hash = 4;
- }
- message Vector {
- repeated Counter counters = 1;
- }
- message Counter {
- uint64 id = 1;
- uint64 value = 2;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields (Index Message)
- .sp
- The \fBfolder\fP field identifies the folder that the index message pertains to.
- .sp
- The \fBfiles\fP field is a list of files making up the index information.
- .SS Fields (FileInfo Message)
- .sp
- The \fBname\fP is the file name path relative to the folder root. Like all
- strings in BEP, the Name is always in UTF\-8 NFC regardless of operating
- system or file system specific conventions. The name field uses the slash
- character (“/”) as path separator, regardless of the implementation’s
- operating system conventions. The combination of folder and name uniquely
- identifies each file in a cluster.
- .sp
- The \fBtype\fP field contains the type of the described item. The type is one
- of \fBfile (0)\fP, \fBdirectory (1)\fP, or \fBsymlink (4)\fP\&.
- .sp
- The \fBsize\fP field contains the size of the file, in bytes. For directories
- and symlinks the size is zero.
- .sp
- The \fBpermissions\fP field holds the common Unix permission bits. An
- implementation MAY ignore or interpret these as is suitable on the host
- operating system.
- .sp
- The \fBmodified_s\fP time is expressed as the number of seconds since the Unix
- Epoch (1970\-01\-01 00:00:00 UTC). The \fBmodified_ns\fP field holds the
- nanosecond part of the modification time.
- .sp
- The \fBmodified_by\fP field holds the short id of the client that last made
- any modification to the file whether add, change or delete. This will be
- overwritten every time a change is made to the file by the last client to do
- so and so does not hold history.
- .sp
- The \fBdeleted\fP field is set when the file has been deleted. The block list
- SHALL be of length zero and the modification time indicates the time of
- deletion or, if the time of deletion is not reliably determinable, the last
- known modification time.
- .sp
- The \fBinvalid\fP field is set when the file is invalid and unavailable for
- synchronization. A peer MAY set this bit to indicate that it can temporarily
- not serve data for the file.
- .sp
- The \fBno_permissions\fP field is set when there is no permission information
- for the file. This is the case when it originates on a file system which
- does not support permissions. Changes to only permission bits SHOULD be
- disregarded on files with this bit set. The permissions bits MUST be set to
- the octal value 0666.
- .sp
- The \fBversion\fP field is a version vector describing the updates performed
- to a file by all members in the cluster. Each counter in the version vector
- is an ID\-Value tuple. The ID is the first 64 bits of the device ID. The
- Value is a simple incrementing counter, starting at zero. The combination of
- Folder, Name and Version uniquely identifies the contents of a file at a
- given point in time.
- .sp
- The \fBsequence\fP field is the value of a device local monotonic clock at the
- time of last local database update to a file. The clock ticks on every local
- database update, thus forming a sequence number over database updates.
- .sp
- The \fBblock_size\fP field is the size, in bytes, of each individual block in
- the block list (except, possibly, the last block). If this field is missing
- or zero, the block size is assumed to be 128 KiB (131072 bytes). Valid
- values of this field are the powers of two from 128 KiB through 16 MiB. See
- also \fI\%Selection of Block Size\fP\&.
- .sp
- The \fBblocks\fP list contains the size and hash for each block in the file.
- Each block represents a \fBblock_size\fP\-sized slice of the file, except for
- the last block which may represent a smaller amount of data. The block list
- is empty for directories and symlinks.
- .sp
- The \fBsymlink_target\fP field contains the symlink target, for entries of
- symlink type. It is empty for all other entry types.
- .SS Request
- .sp
- The Request message expresses the desire to receive a data block
- corresponding to a part of a certain file in the peer’s folder.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Request {
- int32 id = 1;
- string folder = 2;
- string name = 3;
- int64 offset = 4;
- int32 size = 5;
- bytes hash = 6;
- bool from_temporary = 7;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields
- .sp
- The \fBid\fP is the request identifier. It will be matched in the
- corresponding \fBResponse\fP message. Each outstanding request must have a
- unique ID.
- .sp
- The \fBfolder\fP and \fBname\fP fields are as documented for the Index message.
- The \fBoffset\fP and \fBsize\fP fields specify the region of the file to be
- transferred. This SHOULD equate to exactly one block as seen in an Index
- message.
- .sp
- The \fIhash\fP field MAY be set to the expected hash value of the block. If set,
- the other device SHOULD ensure that the transmitted block matches the
- requested hash. The other device MAY reuse a block from a different file and
- offset having the same size and hash, if one exists.
- .sp
- The \fBfrom_temporary\fP field is set to indicate that the read should be
- performed from the temporary file (converting name to it’s temporary form)
- and falling back to the non temporary file if any error occurs. Knowledge of
- contents of temporary files comes from DownloadProgress messages.
- .SS Response
- .sp
- The Response message is sent in response to a Request message.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Response {
- int32 id = 1;
- bytes data = 2;
- ErrorCode code = 3;
- }
- enum ErrorCode {
- NO_ERROR = 0;
- GENERIC = 1;
- NO_SUCH_FILE = 2;
- INVALID_FILE = 3;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields
- .sp
- The \fBid\fP field is the request identifier. It must match the ID of the
- \fBRequest\fP that is being responded to.
- .sp
- The \fBdata\fP field contains either the requested data block or is empty if
- the requested block is not available.
- .sp
- The \fBcode\fP field contains an error code describing the reason a Request
- could not be fulfilled, in the case where zero length data was returned. The
- following values are defined:
- .INDENT 0.0
- .TP
- .B 0
- No Error (data should be present)
- .TP
- .B 1
- Generic Error
- .TP
- .B 2
- No Such File (the requested file does not exist, or the offset is
- outside the acceptable range for the file)
- .TP
- .B 3
- Invalid (file exists but has invalid bit set or is otherwise
- unavailable)
- .UNINDENT
- .SS DownloadProgress
- .sp
- The DownloadProgress message is used to notify remote devices about partial
- availability of files. By default, these messages are sent every 5 seconds,
- and only in the cases where progress or state changes have been detected.
- Each DownloadProgress message is addressed to a specific folder and MUST
- contain zero or more FileDownloadProgressUpdate messages.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message DownloadProgress {
- string folder = 1;
- repeated FileDownloadProgressUpdate updates = 2;
- }
- message FileDownloadProgressUpdate {
- FileDownloadProgressUpdateType update_type = 1;
- string name = 2;
- Vector version = 3;
- repeated int32 block_indexes = 4;
- }
- enum FileDownloadProgressUpdateType {
- APPEND = 0;
- FORGET = 1;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields (DownloadProgress Message)
- .sp
- The \fBfolder\fP field represents the ID of the folder for which the update is
- being provided.
- .sp
- The \fBupdates\fP field is a list of progress update messages.
- .SS Fields (FileDownloadProgressUpdate Message)
- .sp
- The \fBupdate_type\fP indicates whether the update is of type \fBappend (0)\fP
- (new blocks are available) or \fBforget (1)\fP (the file transfer has
- completed or failed).
- .sp
- The \fBname\fP field defines the file name from the global index for which
- this update is being sent.
- .sp
- The \fBversion\fP message defines the version of the file for which this
- update is being sent.
- .sp
- The \fBblock_indexes\fP field is a list of positive integers, where each
- integer represents the index of the block in the FileInfo message Blocks
- array that has become available for download.
- .sp
- For example an integer with value 3 represents that the data defined in the
- fourth BlockInfo message of the FileInfo message of that file is now
- available. Please note that matching should be done on \fBname\fP AND
- \fBversion\fP\&. Furthermore, each update received is incremental, for example
- the initial update message might contain indexes 0, 1, 2, an update 5
- seconds later might contain indexes 3, 4, 5 which should be appended to the
- original list, which implies that blocks 0\-5 are currently available.
- .sp
- Block indexes MAY be added in any order. An implementation MUST NOT assume
- that block indexes are added in any specific order.
- .sp
- The \fBforget\fP field being set implies that previously advertised file is no
- longer available, therefore the list of block indexes should be truncated.
- .sp
- Messages with the \fBforget\fP field set MUST NOT have any block indexes.
- .sp
- Any update message which is being sent for a different \fBversion\fP of the
- same file name must be preceded with an update message for the old version
- of that file with the \fBforget\fP field set.
- .sp
- As a safeguard on the receiving side, the value of \fBversion\fP changing
- between update messages implies that the file has changed and that any
- indexes previously advertised are no longer available. The list of available
- block indexes MUST be replaced (rather than appended) with the indexes
- specified in this message.
- .SS Ping
- .sp
- The Ping message is used to determine that a connection is alive, and to
- keep connections alive through state tracking network elements such as
- firewalls and NAT gateways. A Ping message is sent every 90 seconds, if no
- other message has been sent in the preceding 90 seconds.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Ping {
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Close
- .sp
- The Close message MAY be sent to indicate that the connection will be torn
- down due to an error condition. A Close message MUST NOT be followed by
- further messages.
- .SS Protocol Buffer Schema
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- message Close {
- string reason = 1;
- }
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Fields
- .sp
- The \fBreason\fP field contains a human readable description of the error
- condition.
- .SH SHARING MODES
- .SS Trusted
- .sp
- Trusted mode is the default sharing mode. Updates are exchanged in both
- directions.
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- +\-\-\-\-\-\-\-\-\-\-\-\-+ Updates /\-\-\-\-\-\-\-\-\-\e
- | | \-\-\-\-\-\-\-\-\-\-\-> / \e
- | Device | | Cluster |
- | | <\-\-\-\-\-\-\-\-\-\-\- \e /
- +\-\-\-\-\-\-\-\-\-\-\-\-+ Updates \e\-\-\-\-\-\-\-\-\-/
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Send Only
- .sp
- In send only mode, a device does not apply any updates from the cluster, but
- publishes changes of its local folder to the cluster as usual.
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- +\-\-\-\-\-\-\-\-\-\-\-\-+ Updates /\-\-\-\-\-\-\-\-\-\e
- | | \-\-\-\-\-\-\-\-\-\-\-> / \e
- | Device | | Cluster |
- | | \e /
- +\-\-\-\-\-\-\-\-\-\-\-\-+ \e\-\-\-\-\-\-\-\-\-/
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SS Receive Only
- .sp
- In receive only mode, a device does not send any updates to the cluster, but
- accepts changes to its local folder from the cluster as usual.
- .INDENT 0.0
- .INDENT 3.5
- .sp
- .nf
- .ft C
- +\-\-\-\-\-\-\-\-\-\-\-\-+ Updates /\-\-\-\-\-\-\-\-\-\e
- | | <\-\-\-\-\-\-\-\-\-\-\- / \e
- | Device | | Cluster |
- | | \e /
- +\-\-\-\-\-\-\-\-\-\-\-\-+ \e\-\-\-\-\-\-\-\-\-/
- .ft P
- .fi
- .UNINDENT
- .UNINDENT
- .SH DELTA INDEX EXCHANGE
- .sp
- Index data must be exchanged whenever two devices connect so that one knows
- the files available on the other. In the most basic case this happens by way
- of sending an \fBIndex\fP message followed by one or more \fBIndex Update\fP
- messages. Any previous index data known for a remote device is removed and
- replaced with the new index data received in an \fBIndex\fP message, while the
- contents of an \fBIndex Update\fP message is simply added to the existing
- index data.
- .sp
- For situations with large indexes or frequent reconnects this can be quite
- inefficient. A mechanism can then be used to retain index data between
- connections and only transmit any changes since that data on connection
- start. This is called “delta indexes”. To enable this mechanism the
- \fBsequence\fP and \fBindex ID\fP fields are used.
- .INDENT 0.0
- .TP
- .B Sequence:
- Each index item (i.e., file, directory or symlink) has a sequence number
- field. It contains the value of a counter at the time the index item was
- updated. The counter increments by one for each change. That is, as files
- are scanned and added to the index they get assigned sequence numbers
- 1, 2, 3 and so on. The next file to be changed or detected gets sequence
- number 4, and future updates continue in the same fashion.
- .TP
- .B Index ID:
- Each folder has an Index ID. This is a 64 bit random identifier set at
- index creation time.
- .UNINDENT
- .sp
- Given the above, we know that the tuple {index ID, maximum sequence number}
- uniquely identifies a point in time of a given index. Any further changes
- will increase the sequence number of some item, and thus the maximum
- sequence number for the index itself. Should the index be reset or removed
- (i.e., the sequence number reset to zero), a new index ID must be generated.
- .sp
- By letting a device know the {index ID, maximum sequence number} we have for
- their index data, that device can arrange to only transmit \fBIndex Update\fP
- messages for items with a higher sequence number. This is the delta index
- mechanism.
- .sp
- The index ID and maximum sequence number known for each device is
- transmitted in the \fBCluster Config\fP message at connection start.
- .sp
- For this mechanism to be reliable it is essential that outgoing index
- information is ordered by increasing sequence number. Devices announcing a
- non\-zero index ID in the \fBCluster Config\fP message MUST send all index data
- ordered by increasing sequence number. Devices not intending to participate
- in delta index exchange MUST send a zero index ID or, equivalently, not send
- the \fBindex_id\fP attribute at all.
- .SH MESSAGE LIMITS
- .sp
- An implementation MAY impose reasonable limits on the length of messages and
- message fields to aid robustness in the face of corruption or broken
- implementations. An implementation should strive to keep messages short
- and to the point, favouring more and smaller messages over fewer and larger.
- For example, favour a smaller Index message followed by one or more Index
- Update messages rather than sending a very large Index message.
- .sp
- The Syncthing implementation imposes a hard limit of 500,000,000 bytes on
- all messages. Attempting to send or receive a larger message will result in
- a connection close. This size was chosen to accommodate Index messages
- containing a large block list. It’s intended that the limit may be further
- reduced in a future protocol update supporting variable block sizes (and
- thus shorter block lists for large files).
- .SH SELECTION OF BLOCK SIZE
- .sp
- The desired block size for any given file is the smallest block size that
- results in fewer than 2000 blocks, or the maximum block size for larger
- files. This rule results in the following table of block sizes per file
- size:
- .TS
- center;
- |l|l|.
- _
- T{
- File Size
- T} T{
- Block Size
- T}
- _
- T{
- 0 \- 250 MiB
- T} T{
- 128 KiB
- T}
- _
- T{
- 250 MiB \- 500 MiB
- T} T{
- 256 KiB
- T}
- _
- T{
- 500 MiB \- 1 GiB
- T} T{
- 512 KiB
- T}
- _
- T{
- 1 GiB \- 2 GiB
- T} T{
- 1 MiB
- T}
- _
- T{
- 2 GiB \- 4 GiB
- T} T{
- 2 MiB
- T}
- _
- T{
- 4 GiB \- 8 GiB
- T} T{
- 4 MiB
- T}
- _
- T{
- 8 GiB \- 16 GiB
- T} T{
- 8 MiB
- T}
- _
- T{
- 16 GiB \- up
- T} T{
- 16 MiB
- T}
- _
- .TE
- .sp
- An implementation MAY deviate from the block size rule when there is good
- reason to do so. For example, if a file has been indexed at a certain block
- size and grows beyond 2000 blocks it may be retained at the current block
- size for practical reasons. When there is no overriding reason to the
- contrary, such as when indexing a new file for the first time, the block
- size rule above SHOULD be followed.
- .sp
- An implementation MUST therefore accept files with a block size differing
- from the above rule. This does not mean that arbitrary block sizes are
- allowed. The block size used MUST be exactly one of the power\-of\-two block
- sizes listed in the table above.
- .SH EXAMPLE EXCHANGE
- .TS
- center;
- |l|l|l|.
- _
- T{
- #
- T} T{
- A
- T} T{
- B
- T}
- _
- T{
- 1
- T} T{
- ClusterConfiguration\->
- T} T{
- <\-ClusterConfiguration
- T}
- _
- T{
- 2
- T} T{
- Index\->
- T} T{
- <\-Index
- T}
- _
- T{
- 3
- T} T{
- IndexUpdate\->
- T} T{
- <\-IndexUpdate
- T}
- _
- T{
- 4
- T} T{
- IndexUpdate\->
- T} T{
- T}
- _
- T{
- 5
- T} T{
- Request\->
- T} T{
- T}
- _
- T{
- 6
- T} T{
- Request\->
- T} T{
- T}
- _
- T{
- 7
- T} T{
- Request\->
- T} T{
- T}
- _
- T{
- 8
- T} T{
- Request\->
- T} T{
- T}
- _
- T{
- 9
- T} T{
- T} T{
- <\-Response
- T}
- _
- T{
- 10
- T} T{
- T} T{
- <\-Response
- T}
- _
- T{
- 11
- T} T{
- T} T{
- <\-Response
- T}
- _
- T{
- 12
- T} T{
- T} T{
- <\-Response
- T}
- _
- T{
- 13
- T} T{
- Index Update\->
- T} T{
- T}
- _
- T{
- …
- T} T{
- T} T{
- T}
- _
- T{
- 14
- T} T{
- T} T{
- <\-Ping
- T}
- _
- T{
- 15
- T} T{
- Ping\->
- T} T{
- T}
- _
- .TE
- .sp
- The connection is established and at 1. both peers send ClusterConfiguration
- messages and then Index records. The Index records are received and both
- peers recompute their knowledge of the data in the cluster. In this example,
- peer A has four missing or outdated blocks. At 5 through 8 peer A sends
- requests for these blocks. The requests are received by peer B, who
- retrieves the data from the folder and transmits Response records (9 through
- 12). Device A updates their folder contents and transmits an Index Update
- message (13). Both peers enter idle state after 13. At some later time 14,
- the ping timer on device B expires and a Ping message is sent. The same
- process occurs for device A at 15.
- .SH EXAMPLES OF STRONG CIPHER SUITES
- .TS
- center;
- |l|l|l|.
- _
- T{
- ID
- T} T{
- Name
- T} T{
- Description
- T}
- _
- T{
- 0x009F
- T} T{
- DHE\-RSA\-AES256\-GCM\-SHA384
- T} T{
- TLSv1.2 DH RSA AESGCM(256) AEAD
- T}
- _
- T{
- 0x006B
- T} T{
- DHE\-RSA\-AES256\-SHA256
- T} T{
- TLSv1.2 DH RSA AES(256) SHA256
- T}
- _
- T{
- 0xC030
- T} T{
- ECDHE\-RSA\-AES256\-GCM\-SHA384
- T} T{
- TLSv1.2 ECDH RSA AESGCM(256) AEAD
- T}
- _
- T{
- 0xC028
- T} T{
- ECDHE\-RSA\-AES256\-SHA384
- T} T{
- TLSv1.2 ECDH RSA AES(256) SHA384
- T}
- _
- T{
- 0x009E
- T} T{
- DHE\-RSA\-AES128\-GCM\-SHA256
- T} T{
- TLSv1.2 DH RSA AESGCM(128) AEAD
- T}
- _
- T{
- 0x0067
- T} T{
- DHE\-RSA\-AES128\-SHA256
- T} T{
- TLSv1.2 DH RSA AES(128) SHA256
- T}
- _
- T{
- 0xC02F
- T} T{
- ECDHE\-RSA\-AES128\-GCM\-SHA256
- T} T{
- TLSv1.2 ECDH RSA AESGCM(128) AEAD
- T}
- _
- T{
- 0xC027
- T} T{
- ECDHE\-RSA\-AES128\-SHA256
- T} T{
- TLSv1.2 ECDH RSA AES(128) SHA256
- T}
- _
- .TE
- .SH AUTHOR
- The Syncthing Authors
- .SH COPYRIGHT
- 2014-2019, The Syncthing Authors
- .\" Generated by docutils manpage writer.
- .
|