summary

Rclone is a command line program used to manage files on cloud storage. It is a functional alternative to the cloud provider's network storage interface. More than 40 cloud storage products support rclone, including S3 object storage, enterprise and consumer file storage services, and standard transport protocols.

User name Rclone "Swiss Army Knife for Cloud Storage" 、 "Technology that is no different from magic" 。

Rclone has a powerful cloud equivalent to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support and -- dry run protection. It is used at the command line, script, or through its API.

Rclone has multiple file transfer protocols and supports SFTP, HTTP, WebDAV, FTP and DLNA. Rclone is a mature open source software, initially inspired by rsync and written in Golang. Its documentation and community are also very good, providing a wide range of friendly use cases.

Official website: https://rclone.org/

function

Rclone supports but is not limited to the following functions:

  • Backup (and encrypt) files to cloud storage
  • Recover (and decrypt) files from cloud storage
  • Mirror cloud data to other cloud services or local
  • Migrate data to the cloud or among cloud storage providers
  • Mount multiple, encrypted, cached or diversified cloud storage as disks
  • Use lsf, ljson, size, and ncdu to analyze and describe the data saved on cloud storage
  • Federated file systems present multiple local and/or cloud file systems together as one

characteristic

  • Transfer service

    • Always check MD5, SHA1 hash values to ensure file integrity
    • Timestamp remains in file
    • Operation can be restarted at any time
    • You can access the network, for example, two different cloud providers
    • You can use multithreading to download to the local disk
  • Copy new or changed files to cloud storage
  • Synchronize (one way) to make the directory the same
  • Move files to cloud storage after verification Delete local
  • Check hash and missing/extra files
  • Mount your cloud storage as a network disk
  • Local or remote files via HTTP/WebDav/FTP/SFTP/DLNA services
  • Experimental web-based GUI

use

The official documentation is very detailed. Here we mainly use several application scenarios to see how to use them. The following uses ubuntu22.04 as an example:

install

Rclone is a binary file package that can be downloaded and set environment variables directly for use. Here, the official script is used directly for installation.

 curl -O  https://downloads.rclone.org/rclone-current-linux-amd64.zip unzip rclone-current-linux-amd64.zip cd rclone-*-linux-amd64 sudo cp rclone /usr/bin/ sudo chown root:root /usr/bin/rclone sudo chmod 755 /usr/bin/rclone

Direct execution rclone version You can view the version number and the installation is successful.

to configure

Configuration can be completed step by step by adding configuration files directly or by entering the interactive configuration session command.

The configuration files after the default configuration are saved in: /root/.config/rclone/rclone.conf Directory.

I directly provide Tencent COS configuration file here.

 [tencent cos] # Custom name Type=s3 # Storage type, refer to all supported types in official documents Provider=TencentCOS # provider, refer to official documents or all Env_auth=false # Do not pass the environment variable configuration authentication Access_key_id=AKxxxxxxxx # Key key generated by Tencent Cloud background Secret_access_key=Secretxxxxxxx # The secret key generated in the background of Tencent Cloud Endpoint=cos. ap chengdu. myqcloud. com # Look at the public network address of your bucket in the region where Tencent Cloud cos is located

grammar

 #Local to network disk Rclone [Function Options]<Local Path><Configuration Name: Path>[Parameters] [Parameters] #Network disk to local Rclone [Function Options]<Configuration Name: Path><Local Path>[Parameters] [Parameters] #Online disk to online disk Rclone [Function Options]<Configuration Name: Path><Configuration Name: Path>[Parameters] [Parameters] #[Parameter] is optional

give an example:

 #Synchronize the contents of the local/data/file folder to the/beifen folder under the tencent cos storage, and exclude the file contents specified in/root/excludes.txt rclone sync /data/file tencent-cos:/beifen --exclude-from '/root/excludes.txt' #File synchronization of two network disks Rclone copy configuration network disk name 1: network disk path configuration network disk name 2: network disk path

Command List

use rclone --help You can view all commands. Here, only common commands are listed. For other commands, see the official documents for details:

command explain
rclone copy copy
rclone move If you want to delete the empty source directory after the move, add the -- delete empty src dirs parameter
rclone mount mount
rclone sync Synchronization: synchronize the source directory to the target directory, and only change the target directory
rclone size View the occupied size of the network disk file
rclone delete Delete the file content under the path
rclone purge Delete the path and all its file contents
rclone mkdir Create Directory
rclone rmdir Delete directory
rclone rmdirs Delete the empty directory under the specified environment. If the -- leave root parameter is added, the root directory will not be deleted
rclone check Check whether the source and destination address data match
rclone ls List all files under the specified path, as well as the file size and path
rclone lsl One more display upload time than above
rclone lsd List directories under the specified path
rclone lsf List directories and files under the specified path

Common parameters

command explain
-n = --dry-run Test run to see what rclone will do in actual operation
-P = --progress Display the real-time transmission progress, refresh once every 500 ms, or refresh once every 1 minute by default
--cache-chunk-size 5M The size of the block. By default, the larger the 5M is, the faster the upload will be, and the more memory will be occupied. If the size is too large, the process may be interrupted
--onedrive-chunk-size 100M Improve OneDrive upload speed for G port broadband server
--drive-chunk-size 64M Improving Google Drive upload speed is applicable to G port broadband server
--cache-chunk-total-size SizeSuffix Total size that blocks can occupy on the local disk
--transfers=N The number of parallel files is 4 by default. It is recommended to reduce this parameter on VPS with small memory. For example, it is recommended to set this parameter to 1 on 128M chicken.
--config string Specify the configuration file path, and string is the configuration file path
--ignore-errors Skip Error
--size-only Verify according to the file size, and do not verify hash
--drive-server-side-across-configs Server to server transmission

journal

Rclone has four levels of logging, ERROR ， NOTICE ， INFO and DEBUG 。 By default, rclone generates ERROR and NOTICE Level messages.

  • -q -Rclone will only generate ERROR Message.
  • -v -Rclone will generate ERROR ， NOTICE and INFO Message, Recommend this item 。
  • -vv -Rclone will generate ERROR ， NOTICE ， INFO and DEBUG Message.
  • --log-level LEVEL -Flag controls log level.

filter

If we need to filter out some unimportant files during synchronization, such as program startup logs and sensitive files.

command explain
--exclude-from Exclude file or directory list from a file specification
--include-from Include files or directories from a file specified
--filter-from File filtering rules are specified from a file
--exclude Exclude files or directories
--include Include files or directories
--filter File filtering rules are equivalent to other usage methods of the above two options. Include rules start with+and exclude rules start with -

  • File Type Filtering

    such as --exclude "*.bak" , which means excluding all .bak The ending document can also be written --filter "- *.bak"

    such as --include "*.{png,jpg}" , including all png and jpg Documents, excluding other documents, can also be written --filter "+ *.{png,jpg}"

    --delete-excluded Delete excluded files. It must be used together with the filter parameters, otherwise it is invalid.

  • Directory filtering

    Directory filtering requires adding / , otherwise it will be matched as a file. with / At the beginning, only the root directory (under the specified directory) will be matched, otherwise, the directory will be matched. The same applies to documents.

    --exclude ".git/" Exclude .git catalog.

    --exclude "/.git/" Exclude only those under the root directory .git catalog.

    --exclude "{Video,Software}/" Exclude Video and Software catalog.

    --exclude "/{Video,Software}/" Exclude only those under the root directory Video and Software catalog.

    --include "/{Video,Software}/**" Only include Video and Software All contents of the directory.

  • Size filtering

    The default size unit is kBytes , but can use k ， M or G suffix.

    --min-size Filter files smaller than the specified size. such as --min-size 50 Indicates that files smaller than 50k will not be transferred.

    --max-size Filter files larger than the specified size. such as --max-size 1G Indicates that files larger than 1G will not be transferred.

  • Filter rules come from a file

    --Exclude from<rule file> : Add Include Rule from File

    --Include from<rule file> : Add Delete Rule from File

    --Filter from<rule file> : Add an include/exclude rule from a file. such as --filter-from filter-file.txt 。

    filter-file.txt Example:

     - secret*.jpg + *.jpg + *.png + file2.avi - /dir/Trash/** + /dir/** - *

environment variable

Each option in rclone can be set through the environment variable. The name of the environment variable can be converted through the long option name. Delete -- Prefix, changing - by _ , uppercase with prefix RCLONE_ 。 The priority of the environment variable will be lower than the command line option, that is, the value set by the environment variable will be overwritten when the corresponding option is appended through the command line 。

For example, set the minimum upload size --min-size 50 , using the environment variable is RCLONE_MIN_SIZE=50 。 When the environment variable is set, use the --min-size 100 , then the value of the environment variable will be overwritten by the command line, and the final effect is --min-size 100 。

command explain
RCLONE_CONFIG Custom profile path
RCLONE_CONFIG_PASS If rclone has encryption settings, set this environment variable to password, and the configuration file can be decrypted automatically
RCLONE_RETRIES Number of failed upload retries, 3 times by default
RCLONE_RETRIES_SLEEP The waiting time for upload failure and retry is disabled by default. The units of s, m, and h are seconds, minutes, and hours respectively
CLONE_TRANSFERS Number of files uploaded in parallel
RCLONE_CACHE_CHUNK_SIZE Block size
RCLONE_CACHE_CHUNK_TOTAL_SIZE Total size that blocks can occupy on the local disk
RCLONE_IGNORE_ERRORS=true Skip Error

Best Practices

Backup Local /data/file The directory is stored in Tencent Cloud COS, and scheduled tasks are set to be automatically synchronized every night, which can achieve the effect of regularly backing up important files on the cloud.

  1. catalog /data/file Create under excludes.txt , fill in the list of files to be excluded.

     Logs/* * # Exclude all files under logs *. bak # Exclude files ending in. bak *. zip # Exclude files at the end of. zip

  2. Configure Tencent Cloud COS to ignore here and write directly rclone Command to test whether the execution is synchronized.

     rclone sync -v /data/file tencent-cos:/beifen --exclude-from '/data/file/excludes.txt' >> /root/rclone.log 2>&1
    Manually execute one and check tail -f /root/rclone.log Whether the log has been output. If no error is reported, it will succeed.

  3. Configure scheduled tasks and execute them automatically every day

    implement crontab -e #Open the scheduled task list and jump to the last line to add step 2. The successful command is saved as follows: (it means that synchronization is performed at 3:30 every night)

     30 4 * * * rclone sync -v /data/file tencent-cos:/beifen --exclude-from '/data/file/excludes.txt' >> /root/rclone.log 2>&1

    implement systemctl restart cron.service Restart the scheduled task to take effect

  4. Check after execution time /root/rclone.log The logs under are guaranteed to be OK.

summary

use Rclone It can easily backup synchronized files to all major online disks, simplifying the backup scheme, and can also be used to migrate two different online disk program files. It can even directly attach the online disk to the local disk for expanding disk consumption. For more consumption, please refer to the official documents.

Reference link

Rclone official website

Rclone Advanced Use Tutorial - Common Command Parameters

Article Contents