I must confess that my initial thought for the title was “An optimal repository filestore copy”. Optimal, really ? Relatively to what ? Which variable(s) define(s) the optimality ? Speed\/time to clone ? Too dependent on the installed hardware and software, and the available resources and execution constraints. Simplicity to do it ? Too simple a method can result in a very long execution time while complexity can give a faster solution but be fragile, and vice-versa. Besides, simplicity is a relative concept; a solution may look simple to someone and cause nightmares to some others. Beauty ? I like that one but no, too fuzzy too. Finally, I settled for the present title for it is neutral and up to the point. I leave it up to the reader to judge if the techniques are optimal or simple or beautiful. I only hope that they can be useful to someone.
\nThis article has two parts. In each, I’ll give an alternative for copying a repository’s filestores from one filesystem to another. Actually, both techniques are very similar, they just differ in the way the the content files’ path locations are determined. But let’s start.<\/p>\n
Cloning a Documentum repository is a well-known procedure nowadays. Globally, it implies to first create a placeholder docbase for the clone and then to copy the meta-data stored in the source database, e.g. through an export\/import, usually while the docbase is stopped, plus the document contents, generally stored on disks. If a special storage peripheral is used, such as a Centera CAS or a NAS, there might be a fast, low-level way to clone to content files directly at the device level, check with the manufacturer. If confidentiality is requested and\/or the copy occurs across a network, scp is recommended as it also encrypts the transferred data: Still, if the docbase is large and contains millions to hundreds of millions of documents, copying them to another machine can take some time. If rsync is used, repeated executions will just copy over modified or new documents and optionally remove the deleted ones if the archiving mode is requested (the -a option above) but the first run will take time anyway. Logically, reasonably taking advantage of the available I\/O bandwidth by having several rsync running at once should reduce the time to clone the filestore, shouldn’t it ? Is it possible to apply here a divide-and-conquer technique and process each part simultaneously ? It is, and here is how.<\/p>\n Besides the sheer volume of documents and possibly the limited network and disk I\/O bandwidth, one reason the copy can take a long time, independently from the tools used, is the peculiar way Documentum stores its contents on disk, with all the content files exclusively at the bottom of a 6-level deep sub-tree with the following structure (assuming $my_docbase_root has the same value as above; the letter at column 1 means d for directory and f for file): It should be emphasized that such a particular layout in no way causes a performance penalty in accessing the documents’contents from within the repository because their full path can easily be computed by the content server out of their dm_content.data_ticket attribute (a signed decimal number), e.g. as shown with the one-liner: As there are no files in the intermediate levels, it is necessary to walk the entire tree to reach them and start their processing, which is very time-consuming. Depending on your hardware, a ‘ls -1R’ command can takes hours to complete on a set of fairly dense sub-trees. A contrario, this is an advantage for processing through rsync because rsync is able to create all the necessary sub-path levels (aka “implied directories” in rsync lingo) if the -R|−−relative option is provided, as if a “mkdir -p” were issued; thus, in order to optimally copy an entire filestore, it would be enough to rsync only the terminal directories, once identified, and the whole sub-tree would be recreated implicitly. In the illustration above, the rsync commands for those paths are: With the -R option, we get to transfer only the terminal sub-directories AND their original relative paths, efficiency and convenience ! However, if there are N content files to transfer, at least \u2308N\/256\u2309 such rsync commands will be necessary, e.g. a minimum of 3’900 commands for 1 million files, subject of course to their distribution in the sub-tree (some lesser dense terminal sub-directories can contain less than 256 files so more directories and hence commands are required). It is not documented how the content server distributes them over the sub-directories and there is no balancing job that relocates the content files in order to reduce the number of terminal directories by increasing the density of the left ones and removing the emptied ones. All this is quite sensible because, while it may matter to us, it is a non-issue to the repository. Empty terminal paths can be skipped without fear because they are not referenced in the docbase (only content files are) and hence their absence from the copy cannot introduce inconsistencies.<\/p>\n Of course, in the example above, those hypothetical 3900 rsync commands won’t be launched at once but in groups of some convenient value depending on the load that the machines can endure or the application’s response time degradation if the commands are executed during the normal work hours. Since we are dealing with files and possibly the network, the biggest bottleneck will be the I\/Os and care should be exercised not to saturate them. When dedicated hardware such as high-speed networked NAS with SSD drives are used, this is less of a problem and more parallel rsync instances can be started, but such expensive resources are often shared across a company so that someone might still be impacted at one point. I for one remember one night as I was running one data-intensive DQL query in ten different repositories at once and users were suddenly complaining that they couldn’t work their Word documents any more because response time fell down to a crawl. How was that possible ? What was the relationship between DQL queries in several repositories and a desktop program ? Well, the repositories used Oracle databases whose datafiles were stored on a NAS also used as a host for networked drives mounted on desktop machines. Evidently, that configuration was somewhat sloppy but one never knows how things are configured at a large company, so be prepared for the worst and set up the transfer so that they can be easily suspended, interrupted and resumed. The good thing with rsync in archive mode is that the transfers can be resumed where they left off at a minimal cost just by relaunching the same commands with no need to compute a restarting point.<\/p>\n It goes without saying that setting up public key authentication or ssh connection sharing is mandatory when rsync-ing to a remote machine in order to suppress the thousands of authentication requests that will pop up during the batch execution. <\/p>\n rsync performs mostly I\/Os operations, reading and writing the filesystems, possibly across a network. The time spent in those slow operations (mostly waiting for them to complete, actually) by far outweighs the time spent launching the rsync processes and executing user or system code, especially if the terminal sub-directories are densely filled with large files. Moreover, if the DMClean jobs has been run ahead of time, this 128 * 64K figure is a maximum, it is only reached if all of the terminal sub-directories are not empty. To summarize up to this point, we will address the cost of navigating the special Documentum sub-tree layout by walking the location sub-trees up to the last directory level (or up to 2 levels above if so requested) and generate efficient rsync commands that can easily be parallelized. But before we start, how about asking the repository about its contents ? As it keeps track of it, wouldn’t this alternative be much easier and faster than navigating complex directory sub-trees ? Let’see.<\/p>\n Since a repository obviously knows where its content files are stored on disks, it makes sense to get this information directly from it. In order to be sure to include all the possible renditions as well, we should query dmr_content instead of dm_document(all) (note that the DQL function MFILE_URL() returns those too, so that a “select MFILE_URL(”) from dm_document(all)” could also be used here). Also, unless the dmDMClean job is run beforehand, dmr_content includes orphan contents as well, so this point must be clarified ahead. Anyway, by querying dmr_content we are sure not to omit any content, orphans or not. Line 4 just gets the terminal sub-directories, while line 5 ensures that they are unique in order to avoid rsync-ing the same path multiple times. We use sets here to guarantee distinct terminal path values (set elements are unique in the set).
\nIf all we want is an identical copy of the whole contents’ filesystem, the command dd could be used, e.g. supposing that the source docbase and the clone docbase both use for their contents a dedicated filesystem mounted on \/dev\/mapper\/vol01 respectively on \/dev\/mapper\/vol02:
\n
\nsudo dd if=\/dev\/mapper\/vol01 bs=1024K of=\/dev\/mapper\/vol02
\n<\/code>
\nThe clone’s filesystem \/dev\/mapper\/vol02 could be mounted temporarily on the source docbase’s machine for the copy and later dismounted. If this is not possible, dd can be used over the network, e.g.:
\n
\n# from the clone machine;
\nssh root@source 'dd if=\/dev\/mapper\/vol01 bs=1024K | gzip -1 -' | zcat | dd of=\/dev\/mapper\/vol02
\n
\n# from the source machine as root;
\ndd if=\/dev\/mapper\/vol01 bs=1024K | gzip -1 - | ssh dmadmin@clone 'zcat | dd of=\/dev\/mapper\/vol02'
\n<\/code>
\ndd, and other partition imaging utilities, perform a block by block mirror copy of the source, which is much faster than working at the file level, although it depends on the percentage of used space (if the source filesystem is almost empty, dd will spend most of its time copying unused blocks, which is useless. Here, a simple file by file copy would be more effective).
\nIf it is not possible to work at this low level, e.g. filesystems are not dedicated to repositories’ contents or the types of the filesystems differ, then a file by file copy is required. Modern disks are quite fast, especially for deep-pocket companies using SSD, and so file copy operations should be acceptably quick. A naive command such as the one below could even be used to copy the whole repository’s content (we suppose that the clone will be on the same machine):
\n
\nmy_docbase_root=\/data\/Documentum\/my_docbase
\ndest_path=\/some\/other\/or\/identical\/path\/my_docbase
\ncp -rp ${my_docbase_root} ${dest_path}\/..\/.
\n<\/code>
\nIf $dest_path differs from $my_docbase_root, don’t forget to edit the dm_filestore’s dm_locations of the clone docbase accordingly.<\/p>\n
\n
\nscp -rp ${my_docbase_root} dmadmin@${dest_machine}:${dest_path}\/..\/.
\n<\/code>
\nThe venerable tar command could also be used, on-the-fly and without creating an intermediate archive file, e.g.:
\n
\n( cd ${my_docbase_root}; tar cvzf - * ) | ssh dmadmin@${dest_machine} \"(cd ${dest_path}\/; tar xvzf - )\"
\n<\/code>
\nEven better, the command rsync could be used as it is much more versatile and efficient (and still secure too if configured to use ssh, which is the default), especially if the copy is done live several times in advance during the days preceding the kick-off of the new docbase; such copies will be performed incrementally and will execute quickly, providing an easy way to synchronize the copy with the source. Example of use:
\n
\nrsync -avz --stats ${my_docbase_root}\/ dmadmin@{dest_machine}:${dest_path}
\n<\/code>
\nThe trailing \/ in the source path means copy the content of ${loc_path} but not the directory itself as we assumed it already exists.
\nAlternatively, we can include the directory too:
\n
\nrsync -avz --stats ${my_docbase_root} dmadmin@{dest_machine}:${dest_path}\/..\/.
\n<\/code>
\nIf we run it from the destination machine, the command changes to:
\n
\nrsync -avz --stats dmadmin@source_machine:${my_docbase_root}\/ ${dest_path}\/.
\n<\/code>
\nThe −−stats option is handy to obtain a summary of the transferred files and the resulting performance.<\/p>\nHow Documentum stores the contents on disk<\/h3>\n
\n
\ncd $my_docbase_root
\nd filestore_1
\nd <docbase_id>, e.g. 0000c350
\nd 80 starts with 80 and increases by 1 up to ff, for a total of 2^7 = 128 sub-trees;
\nd 00 up to 16 ^ 16 = 256 directories directly below 80, from 00 to ff
\nd 00 again, up to 256 directories directly below, from 00 to ff, up to 64K total subdirectories at this level; let's call these innermost directories \"terminal directories\";
\nf 00[.ext] files are here, up to 256 files, from 00 to ff per terminal directory
\nf 01[.ext]
\nf ...
\nf ff[.ext]
\nd 01
\nf 00[.ext]
\nf 01[.ext]
\nf ...
\nf ff[.ext]
\nd ...
\nd ff
\nf 00[.ext]
\nf 01[.ext]
\nf ...
\nf ff[.ext]
\nd 01
\nd ...
\nd ff
\nd 81
\nd 00
\nd 01
\nd ...
\nd ff
\nd ...
\nd ff
\nd 00
\nd 01
\nd ...
\nd ff
\nf 00[.ext]
\nf 01[.ext]
\nf ...
\nf ff[.ext]
\n<\/code>
\nThe content files on disk may have an optional extension and are located exclusively at the extremities of their sub-tree; there are no files in the intermediate sub-directories. Said otherwise, the files are the leaves of a filestore directory tree.
\nAll the nodes have a 2-digit lowercase hexadecimal name, from 00 to ff, possibly with holes in the sequences when sub-directories or deleted documents have been swept by the DMClean job. With such a layout, each filestore can store up to (2^7).(2^8).(2^8).(2^8) files, i.e. 2^31 files or a bit more than 2.1 billions files. Any number of filestores can be used for virtually an “unlimited” number of content files. However, since each content object has an 16-digit hexadecimal id whose only last 8 hexadecimal digits are really distinctive (and directly map to the filesystem path of the content file, see below), a docbase can effectively contain “only” up to 16^8 files, i.e. 2^32 content files or slightly more than 4.2 billions files distributed among all the filestores. Too few ? There is hope, aside from spawning new docbases. The knowledge base note here<\/a> explains how “galactic” r_object_id are allocated if more than 4 billions documents are present in a repository, so it should be possible to have literally gazillions of documents in a docbase. It is not clear though whether this galactic concept is implemented yet or whether it has ever been triggered once, so let us stay with our feet firmly on planet Earth for the time being.<\/p>\n
\n
\ndata_ticket=-2147440384
\necho $data_ticket | gawk '{printf(\"%xn\", $0 + 4294967296)}'
\n8000a900
\n<\/code>
\nor, more efficiently entirely in the bash shell:
\n
\nprintf \"%xn\" $(($data_ticket + 4294967296))
\n8000a900
\n<\/code>
\nThis value is now split apart by groups of 2 hex digits with a slash as separator: 80\/00\/a9\/00
\nTo compute the full path, the filestore’s location and the docbase id still need to be prepended to the above partial path, e.g. ${my_docbase_root}\/filestore_01\/0000c350\/80\/00\/a9\/00. Thus, knowing the r_object_id of a document, we can find its content file on the filesystem as shown (or, preferably, using the getpath API function) and knowing the full path of a content file makes it possible to find back the document (or documents as the same content can be shared among several documents) in the repository it belongs to. To be complete, the explanation still needs the concept of filestore and its relationship with a location but let’s stop digressing (check paragraph 3 below for a hint) and focus back to the subject at hand. We have now enough information to get us started.<\/p>\n
\n
\ncd ${my_docbase_root}
\nrsync -avzR --stats filestore_01\/80\/00\/00 dmadmin@{dest_machine}:${dest_path}
\nrsync -avzR --stats filestore_01\/80\/00\/01 dmadmin@{dest_machine}:${dest_path}
\nrsync -avzR --stats filestore_01\/80\/00\/ff dmadmin@{dest_machine}:${dest_path}
\nrsync -avzR --stats filestore_01\/ff\/ff\/00 dmadmin@{dest_machine}:${dest_path}
\n<\/code>
\nIn rsync ≥ v2.6.7, it is even possible to restrict the part within the source full path that should be copied remotely, so no preliminary cd is necessary, e.g.:
\n
\nrsync -avzR --stats ${my_docbase_root}\/.\/<\/strong>filestore_01\/80\/00\/00 dmadmin@{dest_machine}:${dest_path}
\n<\/code>
\nNote the \/.\/ path component, it marks the start of the relative path to reproduce remotely. This command will create the directory ${dest_path}\/filestore_01\/80\/00\/00 on the remote host and copy its content there.
\nPath specification can be quite complicated, so use the -n|−−dry-run and -v|−−verbose (or even -vv for more details) options to have a peek at rsync’s actions before they are applied.<\/p>\n
\nWe potentially replace millions of file by file copy commands with only up to 128 * 64K directory copy commands per filestore, which is much more concise and efficient.<\/p>\n
\nNevertheless, on the bright side, since a tree is by definition acyclic, rsync transfers won’t overlap and therefore can be parallelized without synchronization issues, even when intermediate “implied directories”‘s creations are requested simultaneously by 2 or more rsync commands (if rsync performs an operation equivalent to “mkdir -p”, possible errors due to race conditions during concurrent execution can be simply ignored since the operation is idempotent).<\/p>\nBut up to 128 * 64K rsync commands per filestore : isn’t that a bit too much ?<\/h3>\n
\nStill, rsync has some cleverness of its own in processing the files to copy so why not let it do its job ? Is it possible to reduce their number ? Of course, by just stopping one level before the terminal sub-directories, at their parents’ level. From there, up to 128 * 256 rsync commands are necessary, down from 128 * 64K commands. rsync would then explore itself the up to 64K terminal directories below, hopefully more efficiently than when explicitly told so. For sparse sub-directories or small docbases, this could be more efficient. If so, what would be the cut-off point ? This is a complex question depending on so many factors that there is no other way to answer it than to experiment with several situations. A few informal tests show that copying terminal sub-directories with up to 64K rsync commands is about 40% faster than copying their parent sub-directories. If optimality is defined as speed, then the “64k” variant is the best; if it is defined as compactness, then the “256” variant is the best. One explanation for this could be that the finer and simpler the tasks to perform, the quicker they terminate and free up processes to be reused. Or maybe rsync is overwhelmed by the up to 64K sub-directories to explore and is not so good at that and needs some help. The scripts in this article allow experimenting with the “256” variant.<\/p>\nAlternate solution: just ask the repository !<\/h3>\n
\nThe short python\/bash-like pseudo-code shows how we could do it:<\/p>\n\nfor each filestore in the repository:\n(\n for each content in the filestore:\n get its path, discard its filename;\n add the path to the set of paths to transfer;\n for each path in the set of paths:\n generate an rsync -avzR --stat command;\n) &\n<\/pre>\n
\nLine 7 outputs the rsync commands for the terminal sub-directories and the destination.
\nEven though all filestores are processed concurrently, there could be millions of contents in each filestore and such queries could take forever. However, if we run several such queries in parallel, each working on its own partition (i.e. a non-overlapping subset of the whole such that their union is equal to the whole), we could considerably speed it up. Constraints such “where r_object_id like ‘%0′”, “where r_object_id like ‘%1′”, “where r_object_id like ‘%2′”, .. “where r_object_id like ‘%f'” can slice up the whole set of documents into 16 more or less equally-sized subsets (since the r_object_id is essentially a sequence, its modulo 16 or 256 distribution is uniform), which can then be worked on independently and concurrently. Constraints like “where r_object_id like ‘%00′” .. “where r_object_id like ‘%ff'” can produce 256 slices, and so on.
\nHere is a short python 3 program that does all this:<\/p>\n\n#!\/usr\/bin\/env python\n\n# 12\/2018, C. Cervini, dbi-services;\n \nimport os\nimport sys\nimport traceback\nimport getopt\nfrom datetime import datetime\nimport DctmAPI\nimport multiprocessing\nimport multiprocessing.pool\n\ndef Usage():\n print(\"\"\"Purpose:\nConnects as dmadmin\/xxxx to a given repository and generates rsync commands to transfer the given filestores' contents to the given destination;\nUsage:\n .\/walk_docbase.py -h|--help | -r|--repository [-f|--filestores [{,}]|all] [-d|--dest ] [-l|--level ]\nExample:\n .\/walk_docbase.py --docbase dmtest\nwill list all the filestores in docbase dmtest and exit;\n .\/walk_docbase.py --docbase dmtest --filestore all --dest dmadmin@remote-host:\/documentum\/data\/cloned_dmtest\nwill transfer all the filestores' content to the remote destination in \/documentum\/data, e.g.:\n dm_location.path = \/data\/dctm\/dmtest\/filestore_01 --> \/documentum\/data\/cloned_dmtest\/filestore_01\nif dest does not contain a file path, the same path as the source is used, e.g.:\n .\/walk_docbase.py --docbase dmtest --filestore filestore_01 --dest dmadmin@remote-host\nwill transfer the filestore_01 filestore's content to the remote destination into the same directory, e.g.:\n dm_location.path = \/data\/dctm\/dmtest\/filestore_01 --> \/documentum\/dctm\/dmtest\/filestore_01\nIn any case, the destination root directory, if any is given, must exist as rsync does not create it (although it creates the implied directories);\nGenerated statements can be dry-tested by adding the option --dry-run, e.g.\nrsync -avzR --dry-run \/home\/dmadmin\/documentum\/data\/dmtest\/.\/content_storage_01\/0000c350\/80\/00\/0c dmadmin@dmtest:\/home\/dctm\/dmtest\nCommented out SQL statements to update the dm_location.file_system_path for each dm_filestore are output if the destination path differs from the source's;\nlevel is the starting sub-directory level that will be copied by rsync;\nAllowd values for level are 0 (the default, the terminal directories level), -1 and -2;\nlevel -1 means the sub-directory level above the terminal directories, and so on;\nPractically, use 0 for better granularity and parallelization;\n\"\"\")\n\ndef print_error(error):\n print(error)\n\ndef collect_results(list_paths):\n global stats\n for s in list_paths:\n stats.paths = stats.paths.union(s)\n\ndef explore_fs_slice(stmt):\n unique_paths = set()\n lev = level\n try:\n for item in DctmAPI.select_cor(session, stmt):\n fullpath = DctmAPI.select2dict(session, f\"execute get_path for '{item['r_object_id']}'\")\n last_sep = fullpath[0]['result'].rfind(os.sep)\n fullpath = fullpath[0]['result'][last_sep - 8 : last_sep]\n for lev in range(level, 0):\n last_sep = fullpath.rfind(os.sep)\n fullpath = fullpath[ : last_sep]\n unique_paths.add(fullpath)\n except Exception as e:\n print(e)\n traceback.print_stack()\n DctmAPI.show(f\"for stmt {stmt}, unique_paths={unique_paths}\")\n return unique_paths\n\n# --------------------------------------------------------\n# main;\nif __name__ == \"__main__\":\n DctmAPI.logLevel = 0\n \n # parse the command-line parameters;\n # old-style for more flexibility is not needed here;\n repository = None\n s_filestores = None\n dest = None\n user = \"\"\n dest_host = \"\"\n dest_path = \"\"\n level = 0\n try:\n (opts, args) = getopt.getopt(sys.argv[1:], \"hr:f:d:l:\", [\"help\", \"docbase=\", \"filestore=\", \"destination=\", \"level=\"])\n except getopt.GetoptError:\n print(\"Illegal option\")\n print(\".\/graph-stats.py -h|--help | [-r|--repository ] [-f|--filestores [{,}]|all] [-d|--dest ][-l|--level ]\")\n sys.exit(1)\n for opt, arg in opts:\n if opt in (\"-h\", \"--help\"):\n Usage()\n sys.exit()\n elif opt in (\"-r\", \"--repository\"):\n repository = arg\n elif opt in (\"-f\", \"--filestores\"):\n s_filestores = arg\n elif opt in (\"-d\", \"--dest\"):\n dest = arg\n p_at = arg.rfind(\"@\")\n p_colon = arg.rfind(\":\")\n if -1 != p_at:\n user = arg[ : p_at]\n if -1 != p:colon_\n dest_host = arg[p_at + 1 : p_colon]\n dest_path = arg[p_colon + 1 : ]\n else:\n dest_path = arg[p_at + 1 : ]\n elif -1 != p:colon_\n dest_host = arg[ : p_colon]\n dest_path = arg[p_colon + 1 : ]\n else:\n dest_path = arg\n elif opt in (\"-l\", \"--level\"):\n try:\n level = int(arg)\n if -2 > level or level > 0:\n print(\"raising\")\n raise Exception()\n except:\n print(\"level must be a non positive integer inside the interval (-2, 0)\")\n sys.exit()\n if None == repository:\n print(\"the repository is mandatory\")\n Usage()\n sys.exit()\n if None == dest or \"\" == dest:\n if None != s_filestores:\n print(\"the destination is mandatory\")\n Usage()\n sys.exit()\n if None == s_filestores or 'all' == s_filestores:\n # all filestores requested;\n s_filestores = \"all\"\n filestores = None\n else:\n filestores = s_filestores.split(\",\")\n \n # global parameters;\n # we use a typical idiom to create a cheap namespace;\n class gp: pass\n gp.maxWorkers = 100\n class stats: pass\n\n # connect to the repository;\n DctmAPI.show(f\"Will connect to docbase(s): {repository} and transfer filestores [{s_filestores}] to destination {dest if dest else 'None'}\")\n\n status = DctmAPI.dmInit()\n session = DctmAPI.connect(docbase = repository, user_name = \"dmadmin\", password = \"dmadmin\")\n if session is None:\n print(\"no session opened, exiting ...\")\n exit(1)\n\n # we need the docbase id in hex format;\n gp.docbase_id = \"{:08x}\".format(int(DctmAPI.dmAPIGet(\"get,c,docbaseconfig,r_docbase_id\")))\n\n # get the requested filestores' dm_locations;\n stmt = 'select fs.r_object_id, fs.name, fs.root, l.r_object_id as \"loc_id\", l.file_system_path from dm_filestore fs, dm_location l where {:s}fs.root = l.object_name'.format(f\"fs.name in ({str(filestores)[1:-1]}) and \" if filestores else \"\")\n fs_dict = DctmAPI.select2dict(session, stmt)\n DctmAPI.show(fs_dict)\n if None == dest:\n print(f\"filestores in repository {repository}\")\n for s in fs_dict:\n print(s['name'])\n sys.exit()\n\n # filestores are processed sequentially but inside each filestore, the contents are queried concurrently;\n for storage_id in fs_dict:\n print(f\"# rsync statements for filestore {storage_id['name']};\")\n stats.paths = set()\n stmt = f\"select r_object_id from dmr_content where storage_id = '{storage_id['r_object_id']}' and r_object_id like \"\n a_stmts = []\n for slice in range(16):\n a_stmts.append(stmt + \"'%0{:0x}'\".format(slice))\n gp.path_pool = multiprocessing.pool.Pool(processes = 16)\n gp.path_pool.map_async(explore_fs_slice, a_stmts, len(a_stmts), callback = collect_results, error_callback = print_error)\n gp.path_pool.close()\n gp.path_pool.join()\n SQL_stmts = set()\n for p in stats.paths:\n last_sep = storage_id['file_system_path'].rfind(os.sep)\n if \"\" == dest_path:\n dp = storage_id['file_system_path'][ : last_sep] \n else:\n dp = dest_path\n # note the dot in the source path: relative implied directories will be created from that position; \n print(f\"rsync -avzR {storage_id['file_system_path'][ : last_sep]}\/.{storage_id['file_system_path'][last_sep : ]}\/{str(gp.docbase_id)}\/{p} {(user + '@') if user else ''}{dest_host}{':' if dest_host else ''}{dp}\")\n if storage_id['file_system_path'][ : last_sep] != dest_path:\n # dm_location.file_system_path has changed, give the SQL statements to update them in clone's schema;\n SQL_stmts.add(f\"UPDATE dm_location SET file_system_path = REPLACE(file_system_path, '{storage_id['file_system_path'][ : last_sep]}', '{dest_path}') WHERE r_object_id = '{storage_id['loc_id']}';\")\n # commented out SQL statements to run before starting the repository clone;\n for stmt in SQL_stmts:\n print(f\"# {stmt}\")\n\n status = DctmAPI.disconnect(session)\n if not status:\n print(\"error while disconnecting\")\n<\/pre>\n