def _upload_pg_cluster_dir(self, start_backup_info, pg_cluster_dir,<EOL>version, pool_size, rate_limit=None):
spec, parts = tar_partition.partition(pg_cluster_dir)<EOL>backup_prefix = '<STR_LIT>'.format(self.layout.prefix.rstrip('<STR_LIT:/>'), FILE_STRUCTURE_VERSION,<EOL>**start_backup_info)<EOL>if rate_limit is None:<EOL><INDENT>per_process_limit = None<EOL><DEDENT>else:<EOL><INDENT>per_process_limit = int(rate_limit / pool_size)<EOL><DEDENT>assert per_process_limit is None or per_process_limit > <NUM_LIT:0><EOL>total_size = <NUM_LIT:0><EOL>extended_version_url = backup_prefix + '<STR_LIT>'<EOL>logger.info(<EOL>msg='<STR_LIT>',<EOL>detail=('<STR_LIT>'<EOL>.format(extended_version_url=extended_version_url)))<EOL>uri_put_file(self.creds,<EOL>extended_version_url, BytesIO(version.encode("<STR_LIT:utf8>")),<EOL>content_type='<STR_LIT>')<EOL>logger.info(msg='<STR_LIT>')<EOL>uploader = PartitionUploader(self.creds, backup_prefix,<EOL>per_process_limit, self.gpg_key_id)<EOL>pool = TarUploadPool(uploader, pool_size)<EOL>for tpart in parts:<EOL><INDENT>total_size += tpart.total_member_size<EOL>pool.put(tpart)<EOL><DEDENT>pool.join()<EOL>return spec, backup_prefix, total_size<EOL>
Upload to url_prefix from pg_cluster_dir

This function ignores the directory pg_xlog, which contains WAL
files and are not generally part of a base backup.

Note that this is also lzo compresses the files: thus, the number
of pooled processes involves doing a full sequential scan of the
uncompressed Postgres heap file that is pipelined into lzo. Once
lzo is completely finished (necessary to have access to the file
size) the file is sent to S3 or WABS.

TODO: Investigate an optimization to decouple the compression and
upload steps to make sure that the most efficient possible use of
pipelining of network and disk resources occurs.  Right now it
possible to bounce back and forth between bottlenecking on reading
from the database block device and subsequently the S3/WABS sending
steps should the processes be at the same stage of the upload
pipeline: this can have a very negative impact on being able to
make full use of system resources.

Furthermore, it desirable to overflowing the page cache: having
separate tunables for number of simultanious compression jobs
(which occupy /tmp space and page cache) and number of uploads
(which affect upload throughput) would help.