Content scanning (antivirus)

Syncplify Server! can hand the files that pass through a virtual site to an antivirus or data loss prevention engine and act on its verdict: a file the engine rejects is refused, on every protocol. The engine is asked over ICAP (RFC 3507), the protocol practically every such product offers for this, so no agent is installed on the server and the scanner can live anywhere the server can reach.

The feature is optional and off by default. A site that does not scan pays nothing for it, and a site upgraded from an older version behaves exactly as it did until an operator switches scanning on. It is available with the Professional and Ultimate licenses (and during a trial), starting with version 8.12.

Where it is configured

What	Where
The scanners and the policy of a virtual site	Admin UI, Content scanning (permission Content scanning)
Exempting one account	Admin UI, the user's profile, Skip content scanning
Banning addresses that keep uploading rejected files	Admin UI, Shield, rule Malware uploads (off by default)
What happened to a file	The log (events 4110 to 4114), the notifications of the console, the OnContentRejected event handler

Every virtual site has its own scanners and its own policy. In a cluster, every node that serves the site connects to the scanners itself, so each scanner must be reachable from each node.

How a file is judged

Uploads

Nothing is added while a file is being received. When the upload ends, the server hands the whole file as it now sits on the storage to a scanner, and the verdict decides whether the file is kept or discarded. The client receives its final answer only after the verdict.

Because it is always the whole file that is judged, the same rule covers every way a file can be written: a plain upload, a resumed upload, an append, FTP REST and COMB, writes in the middle of a file. A file that was clean and has something harmful appended to it is refused as a whole.

Protocol	A rejected upload	No scanner could judge it (and the site refuses such files)
SFTP	The close of the file answers a failure with the text the file was rejected by the content scanner, followed by the name the scanner gave	The same, with the text the file could not be scanned and was refused
SCP	The transfer ends with that message and a nonzero exit status	The same
FTP, FTPS, FTPES	550 instead of 226, with that text	451, which tells a client it may try again later
WebClient! and shares	That file is reported as refused (the others of the same upload are not affected), in the user's language	The same

What happens to the storage when an upload is refused depends on what the upload was:

The upload was	When it is refused
A new file	The file is removed.
A replacement of an existing file, on Disk, remote SFTP or R2FS! storage	The file is removed. The previous content is gone, because replacing a file there empties it first. Hold uploads until they are scanned (below) if the previous file must survive.
A replacement of an existing file, on object storage (S3, Azure, Google Cloud Storage)	Nothing is stored. The previous file stays as it was, because an object only comes into existence when the upload ends, after the verdict.
An append or a resumed upload	The file is cut back to what it was before, when everything that was written came after its previous end. Otherwise it is removed.
A WebClient! or share upload	Nothing was written at all: these uploads are judged before they are stored, so a file they would have replaced is still there.

The refused file is removed by the server itself, whatever the account's permissions: an account that may only upload cannot delete, but it does not need to.

Uploads that fail on their own

An upload can also end without ever reaching a verdict: the connection is lost, the client aborts, the quota runs out. What becomes of the part that arrived does not depend on scanning, and is the same on a site that does not scan:

• A WebClient! or share upload arrives in one request and cannot be resumed. The part is removed.
• On storage that cannot continue a file (S3, Azure, Google Cloud Storage, an encrypted disk) the part is discarded. On object storage a file that was being replaced stays as it was.
• Everywhere else the part stays, so that the client can resume it. On a site that scans it is judged before it is kept, like any other file.

A connection that simply drops when the last byte has arrived is not a failure in this sense. Several SFTP clients end every upload that way, so such an upload is treated as complete, and judged.

An FTP client that sends ABOR after its file arrived whole, while the scanner still has it, is answered at once (426, then 226). The file's fate follows the verdict all the same: a clean file is kept, a rejected one is removed.

Downloads

When downloads are scanned, a file is judged before it is served. A file the scanner rejects is refused (SFTP: permission denied, FTP: 550, WebClient!: a message), reported, and left where it is: it may be the only copy of something its owner wants back, so the server never deletes a stored file on its own.

The verdict on a stored file is remembered for as long as the file's size and modification time stay the same (24 hours by default), so a file is read for the scanner once, not once per download. A file judged clean on its way in is not judged again on its first way out. The remembered verdicts are kept per node and can be dropped whenever a scanner reports new definitions.

Download scanning covers what upload scanning cannot: files that reached the storage some other way, files stored before scanning was switched on, and files that were clean by yesterday's definitions.

A request for a file's hash is not a download and is not scanned. A thumbnail in the WebClient! is one: a picture is judged the first time its thumbnail is asked for, so opening a folder of pictures has each of them scanned once, and a rejected picture shows no thumbnail.

ZIP downloads. A download of several files as one archive cannot refuse a single file: the archive is already on its way when that file is reached. A file that is rejected, or that could not be scanned on a site that refuses such files, is left out, and the archive then carries a note as its last entry, _FILES_NOT_INCLUDED.txt, which names every file that is missing and why. The note is in English. It also names files that could not be read for any other reason, and says so when the archive ended at the limit for one download.

While a verdict is pending

Until an upload is judged, nobody may download, rename or truncate that file, including the session that is uploading it. The attempt is refused; it works as soon as the verdict is in. On Disk storage that follows symbolic links this holds for every name that leads to the file: a link to it, and a path through a link to its folder, are refused as well, and an upload made through a link keeps everyone off the file it really writes.

A client that uploads under a temporary name of its own and renames the file when the upload is done (WinSCP does) needs nothing special. On the connection that uploaded the file, the rename is answered only after the verdict: a clean file is renamed, and a rejected one is gone by the time the rename is looked at. An upload that begins at the very moment its file, or the folder it is in, is being renamed is refused, and can simply be tried again.

Holding uploads until they are scanned

On storage that shows a file under its final name from the first byte (local and network disks, encrypted or not, remote SFTP, R2FS!), an upload can be held under a hidden name in the destination folder and published under its real name only once it is judged clean. The setting has three values.

Value	Effect
Off (default)	Uploads are written in place, as in every earlier version, and scanned there.
New files	An upload of a file that does not exist yet is held.
New files and overwrites	An upload that replaces an existing file is held too.

What holding changes:

• A new file does not exist under its name, for anyone, until it is clean. Other users, other nodes of a cluster and programs that watch the folder never see an unjudged or half written file.
• A file that is being replaced keeps serving its previous content until the new content is published, in one step. A rejected replacement leaves the previous file untouched.
• Publishing is a rename inside the same folder. No second copy of the data is made; a replacement occupies space for both versions until it is published.

What to know before switching it on:

• Resuming. A held upload of a new file that is interrupted is discarded, so it cannot be resumed: the client uploads it again. A held replacement that is interrupted is published as the partial file it is (when clean), exactly what an interrupted replacement in place leaves behind, so that a client that resumes continues the right file.
• While the upload runs, asking for the file by name answers "not found" for a new file and describes the previous file for a replacement. Clients that upload under a temporary name of their own and rename afterwards (WinSCP with transfer to temporary file name) are not affected.
• Appends and resumed uploads are never held. They are scanned in place.
• Remote SFTP storage can hold replacements only when the remote server offers the posix-rename@openssh.com extension (OpenSSH does). R2FS! needs a Connector of the release that introduced this feature. When the storage cannot hold an upload, the upload is written in place and scanned there, and event 4114 says so once.
• Object storage (S3, Azure, Google Cloud Storage) needs none of this and ignores the setting: an object only comes into existence when the upload ends, after the verdict.
• On Windows, a held replacement of a file that another program holds open cannot be published; the upload fails with a message and the previous file stays.
• The hidden names (.ssstage- followed by 24 characters and .part) are reserved: no client can create, open, rename, delete or list them, whether or not uploads are held. A leftover from a crash is removed by the server after 24 hours, the next time its folder is listed.

When no scanner can judge a file

Two settings decide about files nobody judged.

A file above the size limit (when a limit is set) is accepted or refused, as chosen. By default there is no limit; with one, larger files are accepted, and each is recorded (event 4111).

No scanner reachable, or no answer in time: the file is refused by default. A site may choose to accept such files instead, and each is then recorded (event 4111).

That choice has a limit, on purpose. Three situations refuse a file whatever the site accepts, because each is something a client can provoke and none says the scanner is merely away:

• the content could not be read back for the scanner (for example a file written out of sequence to object storage),
• the scanner answered with an error for that file,
• the connection to the scanner broke twice while the file was being judged, and the scanner was answering other requests meanwhile. A scanner that finds something and drops the connection looks exactly like this.

A scanner behind a load balancer or a port forwarder does not look unreachable when it is down: its connections are accepted and then closed. The server tells that apart from the case above by asking the scanner a question that carries no file. When that gets no answer either, it is an outage like any other: the scanner is taken out of use, the outage is announced, and the site's choice for an unreachable scanner applies.

A scanner that does not answer is taken out of use after three attempts and tried again every 30 seconds by a single file, so an outage costs one attempt per half minute, not a timeout per file. With several scanners the next one in the list takes over. Events 9080 and 9081 and a notification mark the outage and the return.

A scanner must prove that it scans

A scanner that passes everything looks perfectly healthy, which makes it the most dangerous failure. So before the server uses a scanner, and again whenever the scanner reports new definitions or comes back from an outage, it runs a self test:

1. it asks the scanner what it offers, and checks that the configured method is among it;
2. it sends a small harmless file, which must pass;
3. it sends the antivirus industry's standard test file, which every engine reports by agreement although it is harmless, and which must be rejected.

A scanner that fails is not used (event 9082 and a notification say why) and is tested again every five minutes. The Test button of the Admin UI runs the same test on demand, on a scanner that is saved or one that is still being typed, and shows each step.

A service that is not an antivirus, such as a data loss prevention engine, has no reason to object to the antivirus test file. Its entry has a switch, This service is not an antivirus, that leaves step 3 out. The harmless file must still pass, so a service that refuses everything is still caught; what the server can no longer notice for that entry is a service that lets everything through. The list of scanners and the status panel therefore mark it with Test file check skipped.

Settings

Setting	Default	Meaning
Scan the files of this site	Off	The master switch.
Scanners	none	Up to four. Their order is their priority.
Selection	Failover	Failover always asks the first scanner that is available. Round robin spreads the files over all of them.
Scan	Uploads	Uploads, downloads, or both.
Hold uploads until they are scanned	Off	See above.
Largest file to scan	No limit	In MiB.
A larger file is	Accepted	Or refused.
When no scanner can judge a file	Refuse it	Or accept it. See the limits above.
Excluded paths	none	Paths that are not scanned. Only * and ? are wildcards. An entry with a slash is a path from the root of the user's tree and covers everything below it (/incoming/trusted); an entry without one is a file name and applies in every folder (*.iso). Letter case is ignored. See the two cautions below.
Skip file types the scanner ignores	Off	Do not send the file types a scanner announces it does not look at.
Tell the scanner who is transferring	Off	Sends the client's address and the account name with each file.
Stall timeout	30 seconds	How long an exchange with a scanner may stand still.
Verdict wait	120 seconds	How long a scanner may take to answer once it holds the whole file.
Remember verdicts for	24 hours	For downloads.
Scan again after new definitions	Off	Forget the remembered verdicts when a scanner reports new definitions.
Notify on detection	Off	Raise a notification in the console when a file is rejected (at most one per account every ten minutes).

For each scanner:

Setting	Meaning
Name	Yours to choose, unique within the site.
Address	icap://host:1344/service, or icaps://host:11344/service for TLS. The service name is the one your scanner documents.
Method	RESPMOD is what antivirus engines expect. REQMOD is preferred by some data loss prevention engines.
Certificate authority, Skip verification	For icaps, when the scanner's certificate is not one the system trusts.
Connections	How many connections each node keeps to this scanner for this site, and so how many of its files are scanned at once. Zero means 32, or fewer when the scanner announces a lower limit of its own.
Send files whole	Disables the preview, for a scanner whose preview handling cannot be relied on.
This service is not an antivirus	Leaves the antivirus test file out of the self test.

Two things to know about excluded paths:

• A path is the path as each user sees it, so an entry applies in every user's tree. Excluding /incoming/trusted exempts that folder for every account that has one, including an account that creates it for the purpose. To exempt one account, use its profile (below) instead.
• A file that leaves an excluded path later is not scanned at that moment. A rename or a move is not a transfer of content. A file uploaded as setup.iso and renamed to setup.exe, or uploaded to an excluded folder and moved out of it, has never been judged. Only download scanning looks at it, when someone fetches it.

Exempting an account

Skip content scanning in a user's profile exempts what that account uploads and downloads over its own sessions, for a trusted machine account that moves very large files. It never applies to the account's shares: a share runs as the account that created it, but what arrives through it comes from other people.

What it costs

• A site that does not scan: nothing.
• During a transfer: nothing. The transfer runs at the speed it always did.
• When an upload ends: the time to read the file once more and send it to the scanner, plus the scanner's own time. For local disks and for object storage the file is read locally; for remote SFTP and R2FS! storage it is read back from the remote side.
• Memory: a few hundred kilobytes per file being scanned, whatever its size.

Measured on a development machine with the client, the server and the scanner on the same host, so with no network between them, for SFTP uploads to a local disk:

	64 MiB	256 MiB	1 GiB
Speed while the file is received, with scanning off and on	the same, within the variation between runs	the same	the same
Wait when the upload ends: reading the file back and sending it to a scanner that answers at once	0.03 s	0.1 s	0.4 s
Wait when the upload ends, with ClamAV behind c-icap scanning the file: very regular data	0.2 s	0.8 s	3.6 s
The same, for data that does not compress	2 s	7 s	30 s
Memory in use, more than at rest	about 12 MiB	about 12 MiB	about 12 MiB

Five hundred uploads of 8 MiB at once, against a scanner that takes eight files at a time, all completed, and the scanner never held more than eight. These figures show where the cost falls. They are not what your scanner will need: what a file contains counts for far more than its size. With the same scanner, a ZIP archive of 78 MiB that held 17,000 small files took between one and two minutes, and one of 2 MiB that held 9,000 took close to a minute. A scanner on another machine adds the time it takes to send the file there.

The wait at the end of an upload grows with the file. A scan that takes longer than the verdict wait counts as a file no scanner could judge, so raise that setting on a site whose users send large archives. Clients have their own patience: some give up when the final answer does not come within their timeout, which is usually adjustable in the client. The levers, in the order of their effect, are exempting machine accounts, excluding paths, a size limit, and more scanners or more connections.

Notes for specific scanners

• A file the scanner did not look at is answered like a clean file. Every engine has limits: a largest file, a list of file types, a depth to which it opens archives, archives it cannot open because they are password protected, a time it allows itself. What a product does at its limit, block the file or pass it, is a setting of that product, and a scanner that passes such a file sends the same answer as for a clean one. Neither the server nor its self test can tell the difference. Find your scanner's limits, set them as high as you need, and set the server's Largest file to scan at or below the scanner's own size limit, so that the server knows which files went unscanned and records each of them. The knowledge base article Scanning transfers for malware with an antivirus (ICAP) has the questions to ask of any product and five test uploads that prove what a scanner really looks at.
• c-icap with ClamAV (tested with c-icap 0.5.10, its virus_scan service 0.5.5 and ClamAV 1.4.6, as packaged by Debian 12). As packaged, this pair answers all of the following as clean without looking at them: any file above 5 MiB, through the service name virus_scan; any file whose type is not on the service's list (PDF, PNG and MP4 among others), through the alias avscan; any file above 100 MB; anything inside five or more nested archives; a password protected archive. A file of 2 GiB or more is not scanned whatever the settings, so set Largest file to scan to 2000 with this scanner. Addressed by its plain name, the service also begins to send a file back before it has finished scanning it; a detection then reaches the server as altered content, which it refuses, but without the name of the threat. The knowledge base article Building a free ICAP antivirus scanner with ClamAV and c-icap has the settings that close these, and explains why new signatures must be loaded with c-icap's reconfigure command and not with clamav:dbreload, which resets the engine's limits.
• The number of connections counts across nodes and sites. Every node keeps its own connections to a scanner, for every virtual site that uses it, and many scanners tie up one worker for each open connection, idle or not. Keep connections, times nodes, times sites, within what the scanner serves at once.
• Threat names shown in the log and to clients come from the scanner. Some products send them only when a header option is enabled in their ICAP settings.
• The self test's harmless file and the standard test file are a few dozen bytes each and appear in the scanner's own log as content-scan-probe.txt and antivirus-test-file.com.

Events, notifications and scripts

Event	Meaning
4110 ContentRejected	A scanner rejected a file and the transfer was refused.
4111 ContentAcceptedUnscanned	A file was let through without a verdict.
4112 ContentScanFailed	A file was refused because no scanner could judge it.
4113 RejectedFileCleanupFailed	A refused upload could not be removed and is still on the storage.
4114 StagingUnavailable	Uploads to a virtual file system cannot be held and are scanned in place.
9080 ScanServerUnavailable, 9081 ScanServerRestored	A scanner stopped answering, and came back.
9082 ScanServerMisconfigured	A scanner failed its self test and is not used.
9083 ScanServerTest	An operator tested a scanner.
1015 FeatureNotLicensed	Scanning is switched on but the license does not include it: files are not scanned.

See Log events for their fields. Webhooks deliver these like any other event.

The event handler OnContentRejected fires when a file is rejected, on upload or download. EventCtx() carries RelPath, ScanDirection (upload or download), ScanThreat and ScanServer. It notifies only: the transfer is already refused. A rejected upload also fires OnUploadFail, as every failed upload does, so existing handlers keep working; AfterFileUpload does not fire.

Clusters and licenses

Switching scanning on is refused while any node of the cluster runs a version older than 8.12: such a node would serve files unscanned. The status panel of the Admin UI shows each node's own view of each scanner.

When scanning is switched on and the license no longer includes it, the site does not scan, and says so with event 1015 and a notification. The scanners stay on file.

Limits of this version

• A rejected file is removed, not quarantined.
• The note inside a ZIP download is in English, whatever the language of the WebClient!.
• Appends and resumed uploads cannot be held under a hidden name.
• On storage where uploads are written in place and not held, a server that stops between the last byte and the verdict leaves an unjudged file behind. Scanning downloads covers that case.
• The guard that keeps others off a file whose verdict is pending is per node. Hold uploads to close that gap in a cluster.
• On remote SFTP and R2FS! storage the server cannot tell that two names lead to one file when the remote side has symbolic links, so a file whose verdict is pending can be read there through a link to it. Hold uploads to close that gap on such storage.