Commit Graph
821 Commits
Author SHA1 Message Date
Jakob Borg c48cf7783e wip
Signed-off-by: Jakob Borg <[email protected]>
2026-11-16 08:59:44 +01:00
Jakob Borg be42556744 wip
Signed-off-by: Jakob Borg <[email protected]>
2026-11-16 08:52:30 +01:00
Jakob Borg 2a0e3eb23f wip
Signed-off-by: Jakob Borg <[email protected]>
2026-11-14 13:38:13 +01:00
Jakob Borg d21f613102 wip
Signed-off-by: Jakob Borg <[email protected]>
2026-11-13 07:38:50 +01:00
Jakob Borg 3f795834db feat: Peer-to-peer discovery (fixes #3388)
This implements a new peer-to-peer-based discovery protocol.

- The DHT is based on libp2p (of IPFS etc) and constitutes a new,
  separate Syncthing-specific network. It does not piggyback on any
  existing non-Syncthing nodes out there.
- It uses existing device IDs as the keys, and publishes discovery
  documents that include the device certificate and are signed by the
  same. This enables validation that the discovery data is _for_ the
  right device ID, as well as published _by_ that device.
- The DHT runs on the QUIC socket, so it shares UPnP/NAT-PMP whole
  punches and STUN discovery. (Unless the QUIC listener is disabled,
  then it runs on an ephemeral socket and can effectively do only
  queries and not participate as a full DHT node.)
- There is a hardcoded default pointing to three bootstrap nodes hosted
  in the usual manner.
- Currently it's strictly additive -- it runs separately and in
  parallell with the usual global discovery. Once proven and scaled, it
  will gradually take over the duties of the global discovery.
- There is a new command line tool `stdht` which just runs a DHT node to
  assist with discovery (the bootstrap nodes are this).

The diff is large and terrifying, but that's how it is.

Signed-off-by: Jakob Borg <[email protected]>
2026-11-05 22:43:18 +01:00
Jakob Borg ed38528d7f fix: invalid files on remote devices are not "available" (fixes #10943) (#10944)
Properly exclude them from the corresponding query.

Signed-off-by: Jakob Borg <[email protected]>
2026-10-27 12:25:15 +01:00
Jakob Borg 7eee65afc6 chore(protocol): add protovalidate validation of incoming messages, reasonable limits (#10838)
This adds pre-validation directly at the unmarshal step, of annotations
added to the protobuf definitions. This seeks to weed out unreasonable
situations early, by ensuring offsets are positive, string lengths are
limited, etc. Some of the limits are a bit subjective, but I think I've
put them above what is expected and reasonable...

---------

Signed-off-by: Jakob Borg <[email protected]>
2026-10-18 11:09:03 +02:00
Jakob Borg 5fb192176d fix(sqlite): do not generate unbounded prepared statements (ref #10913) (#10915)
The API can use arbitrary paging parameters, causing arbitrary numbers
of prepared statements, which becomes a memory leak.

Signed-off-by: Jakob Borg <[email protected]>
2026-10-07 19:08:43 +02:00
Jakob Borg 84329b277f fix(sqlite): avoid early garbage collection of deleted recv-enc files (#10916)
Signed-off-by: Jakob Borg <[email protected]>
2026-10-07 19:07:32 +02:00
Jakob Borg 529ced6c6b fix: handle missing max-sequence file entries (#10912)
This could happen when a folder has deleted files as the latest entries,
and nothing else happened for 180 days (the default tombstone cleanup
time).

Signed-off-by: Jakob Borg <[email protected]>
2026-10-07 18:32:44 +02:00
Jakob Borg 8ea09c0094 chore: style fixes from go fix (#10846)
Just `go fix ./...`

Signed-off-by: Jakob Borg <[email protected]>
2026-08-05 20:23:22 +02:00
Jakob Borg 946e2b83a1 fix(db, model): better handle db connections, puller concurrency, avoid deadlock (fixes #10841) (#10842)
There is a danger of deadlock wherever we have database operations
inside a database iterator. The iterator itself pins a connection, so
the nested operations need another connection; if we've reached
maxOpenConns then it blocks until a connection is released. If all
connections are consumed by such iterators then none can make progress.

Luckily, we have limited concurrency for most such iterator loops. They
are part of scanning, pulling, reverting, etc where there is only ever
one such routine per folder. The exception is block reuse in the copier
routine, which is limited by the `Copiers` setting per folder. This
could in practice deadlock since you could set copiers to eight and end
up with six `AllLocalBlocksWithHash` iterators when `maxDBConns=6`, all
of which need to make additional database calls inside the loop.

This PR fixes the problem twice;

- The problematic loop does not need to be reentrant. The set of blocks
that may be returned by the iterator is finite so we can easily just
collect them to a slice before we start processing them. This avoids the
problem entirely.
- We do not need to limit database connections as strictly as we
currently do. Increase the maximum allowed, while reducing the number of
held-open idle connections slightly. This is not an exact science, but
ideally we want "most" operations to be able to use the pinned
connections to avoid cache churn. Most operations are short lived
queries, or single-goroutine iterators with short lived queries inside,
so four connections seems like it should usually be enough. 🤷
- Additionally, set a cap on Copiers. Currently you could set it to an
arbitrarily large number, which is not advantageous. Limit it to
2*NumCPU which scales somewhat with system performance.

Signed-off-by: Jakob Borg <[email protected]>
2026-08-03 23:36:05 +02:00
Jakob Borg 44cbfcad56 fix(sqlite): update last migration to set schema version, counts (#10768)
The counts needs to be modified manually since we're not running
triggers during migrations. The schema version needed to be bumped.

Signed-off-by: Jakob Borg <[email protected]>
2026-06-26 07:22:40 +02:00
Simon Frei c9236b1adc chore: stop treating dirs as having size 128 (#10750)
Resp. directories for the database, and both dirs and symlinks for
`FileInfo.FileSize`. Instead handle it in the UI progress percentage. We
even already have special cases there for deletions, might as well
handle directories just like any other zero-sized needed item there.

This went through the very thorough testing of running it on my laptop,
the migration was applied and it seemed to be working fine after.

---------

Signed-off-by: Simon Frei <[email protected]>
2026-06-22 20:32:03 +02:00
e16b8013a6 chore(db, model): separate methods to drop a device vs its files (#10480)
Fixes #10469

---------

Signed-off-by: Simon Frei <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Co-authored-by: Marcus B Spencer <[email protected]>
Co-authored-by: Jakob Borg <[email protected]>
2026-06-04 18:28:19 +00:00
mattn 8ca3cca0a0 chore: use path/filepath for local file system paths (#10705)
### Purpose

`path` is for slash-separated paths (URLs, BEP protocol); local file
system paths should use `path/filepath`. Fixed in
`cmd/stdiscosrv/database.go` (3 sites) and
`internal/db/sqlite/db_test.go` (1 site).

### Testing

`go build ./cmd/stdiscosrv/...` and `go vet` pass.

Signed-off-by: Yasuhiro Matsumoto <[email protected]>
2026-05-23 22:31:33 +02:00
Jakob Borg 86ac4e5017 feat: make block indexing configurable (#10608)
This adds a new folder-level configuration `FullBlockIndex`. It controls
whether we maintain the block index for a given folder -- currently
that's always true, now it becomes possible to turn off. The block index
is used for lookup of blocks across files and folders. Effectively, when
syncing a change, for each block, we check:

1. Is the block already present in the old version of the file? If so,
we can reuse (copy) it without network transfer. **This check is always
possible.**
2. Is the block already present in any other file in this folder or
other folders? If so we can copy it. **This check is only possible with
the full block index.**
3. We must transfer the block over the network.

Maintaining the full block index is costly in time, I/O and database
size. With this PR, maintaining the full block index becomes the default
for send-receive and receive-only folders only, with it disabled for
send-only and receive-encrypted folders. The block index is never useful
for encrypted folders, as blocks are encrypted separate for each file.
It is also not useful for send-only folders by themselves, though the
data in the send-only folder could be reused by other receive-type
folders if it were enabled.

For very large folders it may make sense to disable the full block index
regardless of folder type and just accept the resulting decrease in data
reuse.

Disabling or enabling the option in the GUI causes the index to be
destroyed or rebuilt accordingly.

https://github.com/syncthing/docs/pull/1005

---------

Signed-off-by: Jakob Borg <[email protected]>
2026-04-26 11:58:09 +02:00
Jakob Borg b39c56f82d chore: remove tracking inode change time (#10579)
Signed-off-by: Jakob Borg <[email protected]>
2026-03-22 20:24:03 -07:00
Jakob Borg 9ffce6e3ff chore(sqlite): reduce max open connections, keep them open permanently (fixes #10592) (#10596)
chore(sqlite): reduce max open connections, keep them open permanently (fixes #10592)

Reduces connection churn, possibly tickling concurrency bug on Windows.

Signed-off-by: Jakob Borg <[email protected]>
2026-03-13 12:03:22 +00:00
Jakob Borg dc2a77ab8e chore: build with Go 1.26; use Go 1.25 features (#10570)
WaitGroup.Go and built-in gomaxprocs handling.

Signed-off-by: Jakob Borg <[email protected]>
2026-02-11 10:41:38 +00:00
Tommy van der VorstandJakob Borg 5cf9168dc2 chore(db): add ability to wait for programmatically started database maintenance, query last maintenance time (#10565)
Also adds a method to query the last database maintenance time.

Signed-off-by: Tommy van der Vorst <[email protected]>
Co-authored-by: Jakob Borg <[email protected]>
2026-02-11 09:28:11 +00:00
Jakob Borg f8a711af4e refactor: remove unused support for Azure blob stores
Signed-off-by: Jakob Borg <[email protected]>
2026-02-04 10:24:17 +01:00
Tommy van der Vorst 5bf27a432c chore(sqlite): allow periodic database maintenance to be disabled (#10441)
This change allows the periodic database maintenance to be disabled, while providing a way to programmatically start maintenance at a convenient moment.

Signed-off-by: Tommy van der Vorst <[email protected]>
2026-01-14 21:10:54 +00:00
Jakob Borg 1843cac6d9 fix(db): remove incorrect comment 2025-12-28 19:33:35 +01:00
Jakob Borg 7c9d06b4d2 chore: linter: embeddedstructfieldcheck
Signed-off-by: Jakob Borg <[email protected]>
2025-10-23 22:48:54 +02:00
Jakob Borg 6f0acacbd2 fix(sqlite): actually always insert blocks for local files (fixes #10388) (#10411)
Due to a thinko, this optimisation was wildly incorrect and would read
to lack of block reuse when syncing files.

(We do not insert a blocklist per device, but only a single one. We
can't use the fact of whether the insert happened as a criteria for
inserting blocks.)

Signed-off-by: Jakob Borg <[email protected]>
2025-09-23 12:46:31 +00:00
Tommy van der Vorst a64c5396e9 fix(db): only perform foreign key checking when a migration was applied (#10397) 2025-09-18 12:22:35 +00:00
Jakob Borg d037681fd1 fix: improve conflict detection by tracking previous file hash (fixes #10349) (#10351)
This adds a new field to the file information we keep, the "previous
blocks hash". This is the hash of the file contents as it was in its
previous incarnation. That is, every scan that updates the blocks hash
will move the current hash to the "previous" field.

This enables an addition to the conflict detection algorithm: if the
file to be synced is in conflict with the current file on disk
(version-counter wise), but it indicates that it was based on the
precise contents we have (new.prevBlocksHash == current.blocksHash),
then it's not really a conflict.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-13 16:16:28 +02:00
Jakob Borg 9ee208b441 chore(sqlite): use normalised tables for file names and versions (#10383)
This changes the files table to use normalisation for the names and
versions. The idea is that these are often common between all remote
devices, and repeating an integer is more efficient than repeating a
long string. A new benchmark bears this out; for a database with 100k
files shared between 31 devices, with some worst case assumption on
version vector size, the database is reduced in size by 50% and the test
finishes quicker:

    Current:
        db_bench_test.go:322: Total size: 6263.70 MiB
    --- PASS: TestBenchmarkSizeManyFilesRemotes (1084.89s)

    New:
        db_bench_test.go:326: Total size: 3049.95 MiB
    --- PASS: TestBenchmarkSizeManyFilesRemotes (776.97s)

The other benchmarks end up about the same within the margin of
variability, with one possible exception being that RemoteNeed seems to
be a little slower on average:

                                          old files/s   new files/s
    Update/n=RemoteNeed/size=1000-8            5.051k        4.654k
    Update/n=RemoteNeed/size=2000-8            5.201k        4.384k
    Update/n=RemoteNeed/size=4000-8            4.943k        4.242k
    Update/n=RemoteNeed/size=8000-8            5.099k        3.527k
    Update/n=RemoteNeed/size=16000-8           3.686k        3.847k
    Update/n=RemoteNeed/size=30000-8           4.456k        3.482k

I'm not sure why, possibly that query can be optimised anyhow.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-12 09:27:41 +00:00
Jakob Borg aa6ae0f3b0 fix(sqlite): add _txlock=immediate to modernc implementation (#10384)
For symmetry with the CGO variant.

https://pkg.go.dev/modernc.org/sqlite#Driver.Open

Signed-off-by: Jakob Borg <[email protected]>
2025-09-11 06:16:31 +00:00
Jakob Borg 25ae01b0d7 chore(sqlite): skip database GC entirely when it's provably unnecessary (#10379)
Store the sequence number of the last GC sweep in a KV. Next time, if it
matches we can just skip GC because nothing has been added or removed.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-08 08:55:04 +02:00
Jakob Borg 4b8d07d91c fix(sqlite): explicitly set temporary directory location (fixes #10368) (#10376)
On Unixes, avoid the /tmp which is likely to become the chosen default.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-07 14:04:47 +02:00
Jakob Borg c33daca3b4 fix(sqlite): less impactful periodic garbage collection (#10374)
Periodic garbage collection can take a long time on large folders. The worst
step is the one for blocks, which are typically orders of magnitude more
numerous than files or block lists.

This improves the situation in by running blocks GC in a number of smaller
range chunks, in random order, and stopping after a time limit. At most ten
minutes per run will be spent garbage collecting blocklists and blocks.

With this, we're not guaranteed to complete a full GC on every run, but
we'll make some progress and get there eventually.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-07 14:04:29 +02:00
Jakob Borg f0180cb014 fix(sqlite): avoid rowid on kv table (#10367)
No migration on this as it has no practical impact, just a slight
cleanup for new installations.

Also a refactor of how we declare single column primary keys, for
consistency.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-05 09:31:07 +00:00
Jakob Borg 36254473a3 chore(slogutil): add configurable logging format (fixes #10352) (#10354)
This adds several options for configuring the log format of timestamps
and severity levels, making it more suitable for integration with log
systems like systemd.

      --log-format-timestamp="2006-01-02 15:04:05"
         Format for timestamp, set to empty to disable timestamps ($STLOGFORMATTIMESTAMP)

      --[no-]log-format-level-string
         Whether to include level string in log line ($STLOGFORMATLEVELSTRING)

      --[no-]log-format-level-syslog
         Whether to include level as syslog prefix in log line ($STLOGFORMATLEVELSYSLOG)

So, to get a timestamp suitable for systemd (syslog prefix, no level
string, no timestamp) we can pass `--log-format-timestamp=""
--no-log-format-level-string --log-format-level-syslog` or,
equivalently, set `STLOGFORMATTIMESTAMP="" STLOGFORMATLEVELSTRING=false
STLOGFORMATLEVELSYSLOG=true`.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-05 10:52:49 +02:00
Jakob Borg 800596139e chore(sqlite): stamp files with application_id
No practical effect, just a tiny bit of fun to stamp the database files
with an application ID that identifies them.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-04 23:15:38 +02:00
Jakob Borg f48782e4df fix(sqlite): revert to default page cache size (#10362)
While we're figuring out optimal defaults, reduce the page cache size to
the compiled-in default. In my computer this makes no difference in
benchmarks. In forum threads, it solved the problem of massive memory
usage during initial scan.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-04 23:07:51 +02:00
Jakob Borg 372e3c26b0 fix(db): remove temp_store = MEMORY pragmas (#10343)
This reduces database migration memory usage in my test scenario from
3.8 GB to 440 MB. In principle I don't think we're causing many temp
tables to be generated anyway in normal usage, but if we do and someone
can benchmark a performance difference, we can add a tunable. I ran the
database benchmark before and after and didn't see a difference above
the noise level.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-03 09:27:53 +02:00
Tommy van der Vorst 6e9ccf7211 fix(db): only vacuum database on startup when a migration script was actually run (#10339) 2025-09-02 12:03:22 -07:00
Jakob Borg 541678ad9e fix(syncthing): apply folder migrations with temporary API/GUI server (#10330)
Prevent the feeling that nothing is happening / it's not starting.

Signed-off-by: Jakob Borg <[email protected]>
2025-09-01 22:10:48 +02:00
Jakob Borg e41d6b9c1e fix(db): apply all migrations and schema in one transaction 2025-08-31 12:43:41 +02:00
Jakob Borg 21ad99c80a Revert "chore(db): update schema version in the same transaction as migration (#10321)"
This reverts commit 4459438245.
2025-08-31 12:43:41 +02:00
Jakob Borgandbt90 4ad3f07691 chore(db): migration for previous commits (#10319)
Recreate the blocks and block lists tables.

---------

Co-authored-by: bt90 <[email protected]>
2025-08-31 09:27:33 +02:00
Simon Frei 4459438245 chore(db): update schema version in the same transaction as migration (#10321)
Just to be entirely sure that if the migration succeeds the schema
version is always also updated. Currently if a migration succeeds but a
later migration doesn't, the changes of the migration apply but the
version stays - if the migration is breaking/non-idempotent, it will
fail when it tries to rerun it next time (otherwise it's just a
pointless re-execution).

Unfortunately with the current `db.runScripts` it wasn't that easy to
do, so I had to do quite a bit of refactoring. I am also ensuring the
right order of transactions now, though I assume that was already the
case lexicographically - can't hurt to be safe.
2025-08-30 13:18:31 +02:00
Jakob Borg 2306c6d989 chore(db): benchmark output, migration blocks/s output (#10320)
Just minor tweaks
2025-08-29 14:58:38 +00:00
Jakob Borg c918299eab refactor(db): slightly improve insert performance (#10318)
This just removes an unnecessary foreign key constraint, where we
already do the garbage collection manually in the database service.
However, as part of getting here I tried a couple of other variants
along the way:

- Changing the order of the primary key from `(hash, blocklist_hash,
idx)` to `(blocklist_hash, idx, hash)` so that inserts would be
naturally ordered. However this requires a new index `on blocks (hash)`
so that we can still look up blocks by hash, and turns out to be
strictly worse than what we already have.
- Removing the primary key entirely and the `WITHOUT ROWID` to make it a
rowid table without any required order, and an index as above. This is
faster when the table is small, but becomes slower when it's large (due
to dual indexes I guess).

These are the benchmark results from current `main`, the second
alternative below ("Index(hash)") and this proposal that retains the
combined primary key ("combined"). Overall it ends up being about 65%
faster.

<img width="764" height="452" alt="Screenshot 2025-08-29 at 14 36 28"
src="https://github.com/user-attachments/assets/bff3f9d1-916a-485f-91b7-b54b477f5aac"
/>

Ref #10264
2025-08-29 15:26:23 +02:00
bt90 b59443f136 chore(db): avoid rowid for blocks and blocklists (#10315)
### Purpose

Noticed "large" autgenerated indices on blocks and blocklists in
https://forum.syncthing.net/t/database-or-disk-is-full-might-be-syncthing-might-be-qnap/24930/7

They both have a primary key and don't need rowids

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
2025-08-29 11:12:39 +02:00
Simon Frei e54f51c9c5 chore(db): cleanup DB in tests and remove OpenTemp (#10282)
Filled up my tmpfs with test DBs when running benchmarks :)
2025-08-24 09:58:56 +00:00
Simon Frei a259a009c8 chore(db): adjust db bench name to improve benchstat grouping (#10283)
The benchstat tool allows custom grouping when comparing with what it
calls "sub-name configuration keys":

https://pkg.go.dev/golang.org/x/[email protected]/cmd/benchstat#hdr-Configuring_comparisons

That's quite useful for these benchmarks, as we basically have two
independent configs: The type of benchmark and the size. Real example
usage for the prepared named statements PR (results are rubbish for
unrelated reasons):

```
$ benchstat -row ".name /n" bench-main.out bench-prepared.out
goos: linux
goarch: amd64
pkg: github.com/syncthing/syncthing/internal/db/sqlite
cpu: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
                            │ bench-main-20250823_014059.out │   bench-prepared-20250823_022849.out   │
                            │             sec/op             │     sec/op      vs base                │
Update Insert100Loc                           248.5m ±  8% ¹   157.7m ±  7% ¹  -36.54% (p=0.000 n=50)
Update RepBlocks100                           253.7m ±  4% ¹   163.6m ±  7% ¹  -35.49% (p=0.000 n=50)
Update RepSame100                            130.42m ±  3% ¹   60.26m ±  2% ¹  -53.80% (p=0.000 n=50)
Update Insert100Rem                           38.54m ±  5% ¹   21.94m ±  1% ¹  -43.07% (p=0.000 n=50)
Update GetGlobal100                          10.897m ±  4% ¹   4.231m ±  1% ¹  -61.17% (p=0.000 n=50)
Update LocalSequenced                         7.560m ±  5% ¹   3.124m ±  2% ¹  -58.68% (p=0.000 n=50)
Update GetDeviceSequenceLoc                  17.554µ ±  6% ¹   8.400µ ±  1% ¹  -52.15% (n=50)
Update GetDeviceSequenceRem                  17.727µ ±  4% ¹   8.237µ ±  2% ¹  -53.54% (p=0.000 n=50)
Update RemoteNeed                              4.147 ± 77% ¹    1.903 ± 78% ¹  -54.11% (p=0.000 n=50)
Update LocalNeed100Largest                   21.516m ± 22% ¹   9.312m ± 47% ¹  -56.72% (p=0.000 n=50)
geomean                                       15.35m           7.486m          -51.22%
¹ benchmarks vary in .fullname
```
2025-08-23 16:12:55 +02:00
Jakob Borg 8151bcddff fix(db): clean files for dropped folders at startup (#10280)
This adds a cleanup stage to remove database files for folders that no
longer exist on startup. Folder database files were already removed when
dropping a folder, assuming that the folder database had been opened at
that point. This won't be the case though when a folder is removed from
the config when Syncthing isn't running, or when a folder is dropped and
re-migrated in a restarted migration.
2025-08-22 09:00:05 +02:00