Add an option to upload incomplete cloning to S3

Add more logging to show successful/failed counts
thevickypedia · Nov 3, 2024 · 481e9cb · 481e9cb
1 parent 95beb5d
commit 481e9cb
Show file tree

Hide file tree

Showing 12 changed files with 84 additions and 43 deletions.
diff --git a/README.md b/README.md
@@ -64,6 +64,7 @@ git2s3 start
 - **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
 - **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **INCOMPLETE_UPLOAD** - Boolean flag to upload incomplete cloning. Defaults to `False`
 - **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
 - **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
 - **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.

diff --git a/docs/README.html b/docs/README.html
@@ -4,7 +4,7 @@
 <html lang="en">
   <head>
     <meta charset="utf-8" />
-    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
 
     <title>Git2S3 &#8212; Git2S3  documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
@@ -100,6 +100,7 @@ <h2>Environment Variables<a class="headerlink" href="#environment-variables" tit
 <li><p><strong>LOG</strong> - Log options to log to a <code class="docutils literal notranslate"><span class="pre">file</span></code> or <code class="docutils literal notranslate"><span class="pre">stdout</span></code>. <em>Does not apply when custom logger is used</em></p></li>
 <li><p><strong>DEBUG</strong> - Boolean flag to enable debug level logging. <em>Does not apply when custom logger is used</em></p></li>
 <li><p><strong>STORE_LOCAL</strong> - Boolean flag to store the backup locally. Defaults to <code class="docutils literal notranslate"><span class="pre">False</span></code></p></li>
+<li><p><strong>INCOMPLETE_UPLOAD</strong> - Boolean flag to upload incomplete cloning. Defaults to <code class="docutils literal notranslate"><span class="pre">False</span></code></p></li>
 <li><p><strong>AWS_PROFILE_NAME</strong> - AWS profile name. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_DEFAULT_PROFILE</span></code> by default.</p></li>
 <li><p><strong>AWS_ACCESS_KEY_ID</strong> - AWS access key ID. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_ACCESS_KEY_ID</span></code> by default.</p></li>
 <li><p><strong>AWS_SECRET_ACCESS_KEY</strong> - AWS secret key. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_SECRET_ACCESS_KEY</span></code> by default.</p></li>

diff --git a/docs/README.md b/docs/README.md
@@ -64,6 +64,7 @@ git2s3 start
 - **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
 - **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **INCOMPLETE_UPLOAD** - Boolean flag to upload incomplete cloning. Defaults to `False`
 - **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
 - **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
 - **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.

diff --git a/docs/_sources/README.md.txt b/docs/_sources/README.md.txt
@@ -64,6 +64,7 @@ git2s3 start
 - **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
 - **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **INCOMPLETE_UPLOAD** - Boolean flag to upload incomplete cloning. Defaults to `False`
 - **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
 - **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
 - **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.

diff --git a/docs/genindex.html b/docs/genindex.html
@@ -231,10 +231,12 @@ <h2 id="H">H</h2>
 <h2 id="I">I</h2>
 <table style="width: 100%" class="indextable genindextable"><tr>
   <td style="width: 33%; vertical-align: top;"><ul>
-      <li><a href="index.html#git2s3.exc.InvalidOwner">InvalidOwner</a>
+      <li><a href="index.html#git2s3.config.EnvConfig.incomplete_upload">incomplete_upload (git2s3.config.EnvConfig attribute)</a>
 </li>
   </ul></td>
   <td style="width: 33%; vertical-align: top;"><ul>
+      <li><a href="index.html#git2s3.exc.InvalidOwner">InvalidOwner</a>
+</li>
       <li><a href="index.html#git2s3.exc.InvalidSource">InvalidSource</a>
 </li>
   </ul></td>

diff --git a/docs/index.html b/docs/index.html
@@ -4,7 +4,7 @@
 <html lang="en">
   <head>
     <meta charset="utf-8" />
-    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
 
     <title>Welcome to Git2S3’s documentation! &#8212; Git2S3  documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
@@ -96,7 +96,7 @@ <h1>Welcome to Git2S3’s documentation!<a class="headerlink" href="#welcome-to-
 
 <dl class="py method">
 <dt class="sig sig-object py" id="git2s3.main.Git2S3.cli">
-<span class="sig-name descname"><span class="pre">cli</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">cmd</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">fail</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">int</span></span></span><a class="headerlink" href="#git2s3.main.Git2S3.cli" title="Permalink to this definition">¶</a></dt>
+<span class="sig-name descname"><span class="pre">cli</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">cmd</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">fail</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">retry</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">int</span></span></span><a class="headerlink" href="#git2s3.main.Git2S3.cli" title="Permalink to this definition">¶</a></dt>
 <dd><p>Runs CLI commands.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -251,14 +251,14 @@ <h1>Welcome to Git2S3’s documentation!<a class="headerlink" href="#welcome-to-
 
 <dl class="py method">
 <dt class="sig sig-object py" id="git2s3.s3.Uploader.trigger">
-<span class="sig-name descname"><span class="pre">trigger</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">bool</span></span></span><a class="headerlink" href="#git2s3.s3.Uploader.trigger" title="Permalink to this definition">¶</a></dt>
+<span class="sig-name descname"><span class="pre">trigger</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">int</span></span></span><a class="headerlink" href="#git2s3.s3.Uploader.trigger" title="Permalink to this definition">¶</a></dt>
 <dd><p>Trigger to upload all file objects concurrently to S3.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Returns<span class="colon">:</span></dt>
-<dd class="field-odd"><p>Returns a boolean flag to indicate completion status.</p>
+<dd class="field-odd"><p>Returns a failed count to indiciate the number files that were failed to upload.</p>
 </dd>
 <dt class="field-even">Return type<span class="colon">:</span></dt>
-<dd class="field-even"><p>bool</p>
+<dd class="field-even"><p>int</p>
 </dd>
 </dl>
 </dd></dl>
@@ -341,17 +341,17 @@ <h1>Welcome to Git2S3’s documentation!<a class="headerlink" href="#welcome-to-
 
 <dl class="py function">
 <dt class="sig sig-object py" id="git2s3.squire.check_file_presence">
-<span class="sig-prename descclassname"><span class="pre">git2s3.squire.</span></span><span class="sig-name descname"><span class="pre">check_file_presence</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">source_dir</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">os.PathLike</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">bool</span></span></span><a class="headerlink" href="#git2s3.squire.check_file_presence" title="Permalink to this definition">¶</a></dt>
+<span class="sig-prename descclassname"><span class="pre">git2s3.squire.</span></span><span class="sig-name descname"><span class="pre">check_file_presence</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">source_dir</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">os.PathLike</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">int</span></span></span><a class="headerlink" href="#git2s3.squire.check_file_presence" title="Permalink to this definition">¶</a></dt>
 <dd><p>Get a list of all subdirectories and check for file presence.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>
 <dd class="field-odd"><p><strong>source_dir</strong> – Root directory to check for file presence.</p>
 </dd>
 <dt class="field-even">Returns<span class="colon">:</span></dt>
-<dd class="field-even"><p>Returns a bool indicating if files are present in the subdirectories.</p>
+<dd class="field-even"><p>Returns the total number of zip files cloned.</p>
 </dd>
 <dt class="field-odd">Return type<span class="colon">:</span></dt>
-<dd class="field-odd"><p>bool</p>
+<dd class="field-odd"><p>int</p>
 </dd>
 </dl>
 </dd></dl>
@@ -421,6 +421,11 @@ <h1>Configuration<a class="headerlink" href="#configuration" title="Permalink to
 <span class="sig-name descname"><span class="pre">git_ignore</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#git2s3.config.EnvConfig.git_ignore" title="Permalink to this definition">¶</a></dt>
 <dd></dd></dl>
 
+<dl class="py attribute">
+<dt class="sig sig-object py" id="git2s3.config.EnvConfig.incomplete_upload">
+<span class="sig-name descname"><span class="pre">incomplete_upload</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#git2s3.config.EnvConfig.incomplete_upload" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
 <dl class="py attribute">
 <dt class="sig sig-object py" id="git2s3.config.EnvConfig.source">
 <span class="sig-name descname"><span class="pre">source</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#git2s3.config.SourceControl" title="git2s3.config.SourceControl"><span class="pre">SourceControl</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#git2s3.config.SourceControl" title="git2s3.config.SourceControl"><span class="pre">SourceControl</span></a><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#git2s3.config.EnvConfig.source" title="Permalink to this definition">¶</a></dt>

diff --git a/docs/objects.inv b/docs/objects.inv