Set .wiki as suffix for wiki pages after archiving

Create new env var to store a local copy of the backup
Remove cloned directory after successful S3 upload (default)

README.md

@@ -61,12 +61,13 @@ git2s3 start
 - **GIT_TOKEN** - GitHub token to get ALL repos (including private).
 - **GIT_IGNORE** - List of repositories/gists to ignore. Defaults to `[]`
 - **SOURCE** - Source options `[repo, gist, wiki]` to back up. Defaults to all.
-- **LOG** - Log options to log to a ``file`` or ``stdout``. _Does not apply when custom logger is used_
+- **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
-- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value ``AWS_DEFAULT_PROFILE`` by default.
-- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value ``AWS_ACCESS_KEY_ID`` by default.
-- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value ``AWS_SECRET_ACCESS_KEY`` by default.
-- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value ``AWS_DEFAULT_REGION`` by default.
+- **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
+- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
+- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.
+- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value `AWS_DEFAULT_REGION` by default.
 - **AWS_BUCKET_NAME** - AWS bucket name to store the backups.
 - **AWS_S3_PREFIX** - S3 prefix _(folder like)_ for the backup. Defaults to `github`
 - **BOTO3_RETRY_ATTEMPTS** - Number of retries for Boto3 client config. Defaults to `10`

docs/README.html

@@ -99,6 +99,7 @@ <h2>Environment Variables<a class="headerlink" href="#environment-variables" tit
 <li><p><strong>SOURCE</strong> - Source options <code class="docutils literal notranslate"><span class="pre">[repo,</span> <span class="pre">gist,</span> <span class="pre">wiki]</span></code> to back up. Defaults to all.</p></li>
 <li><p><strong>LOG</strong> - Log options to log to a <code class="docutils literal notranslate"><span class="pre">file</span></code> or <code class="docutils literal notranslate"><span class="pre">stdout</span></code>. <em>Does not apply when custom logger is used</em></p></li>
 <li><p><strong>DEBUG</strong> - Boolean flag to enable debug level logging. <em>Does not apply when custom logger is used</em></p></li>
+<li><p><strong>STORE_LOCAL</strong> - Boolean flag to store the backup locally. Defaults to <code class="docutils literal notranslate"><span class="pre">False</span></code></p></li>
 <li><p><strong>AWS_PROFILE_NAME</strong> - AWS profile name. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_DEFAULT_PROFILE</span></code> by default.</p></li>
 <li><p><strong>AWS_ACCESS_KEY_ID</strong> - AWS access key ID. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_ACCESS_KEY_ID</span></code> by default.</p></li>
 <li><p><strong>AWS_SECRET_ACCESS_KEY</strong> - AWS secret key. Uses the CLI config value <code class="docutils literal notranslate"><span class="pre">AWS_SECRET_ACCESS_KEY</span></code> by default.</p></li>

docs/README.md

@@ -61,12 +61,13 @@ git2s3 start
 - **GIT_TOKEN** - GitHub token to get ALL repos (including private).
 - **GIT_IGNORE** - List of repositories/gists to ignore. Defaults to `[]`
 - **SOURCE** - Source options `[repo, gist, wiki]` to back up. Defaults to all.
-- **LOG** - Log options to log to a ``file`` or ``stdout``. _Does not apply when custom logger is used_
+- **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
-- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value ``AWS_DEFAULT_PROFILE`` by default.
-- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value ``AWS_ACCESS_KEY_ID`` by default.
-- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value ``AWS_SECRET_ACCESS_KEY`` by default.
-- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value ``AWS_DEFAULT_REGION`` by default.
+- **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
+- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
+- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.
+- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value `AWS_DEFAULT_REGION` by default.
 - **AWS_BUCKET_NAME** - AWS bucket name to store the backups.
 - **AWS_S3_PREFIX** - S3 prefix _(folder like)_ for the backup. Defaults to `github`
 - **BOTO3_RETRY_ATTEMPTS** - Number of retries for Boto3 client config. Defaults to `10`

docs/_sources/README.md.txt

@@ -61,12 +61,13 @@ git2s3 start
 - **GIT_TOKEN** - GitHub token to get ALL repos (including private).
 - **GIT_IGNORE** - List of repositories/gists to ignore. Defaults to `[]`
 - **SOURCE** - Source options `[repo, gist, wiki]` to back up. Defaults to all.
-- **LOG** - Log options to log to a ``file`` or ``stdout``. _Does not apply when custom logger is used_
+- **LOG** - Log options to log to a `file` or `stdout`. _Does not apply when custom logger is used_
 - **DEBUG** - Boolean flag to enable debug level logging. _Does not apply when custom logger is used_
-- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value ``AWS_DEFAULT_PROFILE`` by default.
-- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value ``AWS_ACCESS_KEY_ID`` by default.
-- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value ``AWS_SECRET_ACCESS_KEY`` by default.
-- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value ``AWS_DEFAULT_REGION`` by default.
+- **STORE_LOCAL** - Boolean flag to store the backup locally. Defaults to `False`
+- **AWS_PROFILE_NAME** - AWS profile name. Uses the CLI config value `AWS_DEFAULT_PROFILE` by default.
+- **AWS_ACCESS_KEY_ID** - AWS access key ID. Uses the CLI config value `AWS_ACCESS_KEY_ID` by default.
+- **AWS_SECRET_ACCESS_KEY** - AWS secret key. Uses the CLI config value `AWS_SECRET_ACCESS_KEY` by default.
+- **AWS_REGION_NAME** - S3 bucket's region. Uses the CLI config value `AWS_DEFAULT_REGION` by default.
 - **AWS_BUCKET_NAME** - AWS bucket name to store the backups.
 - **AWS_S3_PREFIX** - S3 prefix _(folder like)_ for the backup. Defaults to `github`
 - **BOTO3_RETRY_ATTEMPTS** - Number of retries for Boto3 client config. Defaults to `10`

docs/genindex.html

@@ -243,10 +243,12 @@ <h2 id="I">I</h2>
 <h2 id="L">L</h2>
 <table style="width: 100%" class="indextable genindextable"><tr>
   <td style="width: 33%; vertical-align: top;"><ul>
-      <li><a href="index.html#git2s3.config.EnvConfig.log">log (git2s3.config.EnvConfig attribute)</a>
+      <li><a href="index.html#git2s3.config.EnvConfig.local_store">local_store (git2s3.config.EnvConfig attribute)</a>
 </li>
   </ul></td>
   <td style="width: 33%; vertical-align: top;"><ul>
+      <li><a href="index.html#git2s3.config.EnvConfig.log">log (git2s3.config.EnvConfig attribute)</a>
+</li>
       <li><a href="index.html#git2s3.config.LogOptions">LogOptions (class in git2s3.config)</a>
 </li>
   </ul></td>

docs/index.html

@@ -251,8 +251,16 @@ <h1>Welcome to Git2S3’s documentation!<a class="headerlink" href="#welcome-to-
 
 <dl class="py method">
 <dt class="sig sig-object py" id="git2s3.s3.Uploader.trigger">
-<span class="sig-name descname"><span class="pre">trigger</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="headerlink" href="#git2s3.s3.Uploader.trigger" title="Permalink to this definition">¶</a></dt>
+<span class="sig-name descname"><span class="pre">trigger</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">bool</span></span></span><a class="headerlink" href="#git2s3.s3.Uploader.trigger" title="Permalink to this definition">¶</a></dt>
 <dd><p>Trigger to upload all file objects concurrently to S3.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns<span class="colon">:</span></dt>
+<dd class="field-odd"><p>Returns a boolean flag to indicate completion status.</p>
+</dd>
+<dt class="field-even">Return type<span class="colon">:</span></dt>
+<dd class="field-even"><p>bool</p>
+</dd>
+</dl>
 </dd></dl>
 
 </dd></dl>
@@ -428,6 +436,11 @@ <h1>Configuration<a class="headerlink" href="#configuration" title="Permalink to
 <span class="sig-name descname"><span class="pre">debug</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#git2s3.config.EnvConfig.debug" title="Permalink to this definition">¶</a></dt>
 <dd></dd></dl>
 
+<dl class="py attribute">
+<dt class="sig sig-object py" id="git2s3.config.EnvConfig.local_store">
+<span class="sig-name descname"><span class="pre">local_store</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">bool</span></em><a class="headerlink" href="#git2s3.config.EnvConfig.local_store" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
 <dl class="py attribute">
 <dt class="sig sig-object py" id="git2s3.config.EnvConfig.aws_profile_name">
 <span class="sig-name descname"><span class="pre">aws_profile_name</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#git2s3.config.EnvConfig.aws_profile_name" title="Permalink to this definition">¶</a></dt>

git2s3/__init__.py

@@ -6,7 +6,7 @@
 
 from git2s3.main import Git2S3
 
-version = "0.0.2"
+version = "0.0.3"
 
 
 @click.command()

git2s3/config.py

@@ -81,6 +81,7 @@ class EnvConfig(BaseSettings):
     source: SourceControl | List[SourceControl] = SourceControl.all
     log: LogOptions = LogOptions.stdout
     debug: bool = False
+    local_store: bool = False
 
     aws_profile_name: str | None = None
     aws_access_key_id: str | None = None

git2s3/main.py

@@ -217,12 +217,17 @@ def clone_wiki(self, datastore: config.DataStore) -> None:
         if datastore.private:
             wiki_dest = str(
                 os.path.join(
-                    self.clone_dir, datastore.source, "private", datastore.name
+                    self.clone_dir,
+                    datastore.source,
+                    "private",
+                    f"{datastore.name}.wiki",
                 )
             )
         else:
             wiki_dest = str(
-                os.path.join(self.clone_dir, datastore.source, "public", datastore.name)
+                os.path.join(
+                    self.clone_dir, datastore.source, "public", f"{datastore.name}.wiki"
+                )
             )
         if not os.path.isdir(wiki_dest):
             os.makedirs(wiki_dest)
@@ -376,7 +381,12 @@ def start(self) -> None:
         if squire.check_file_presence(self.clone_dir):
             self.logger.info("Initiating S3 upload process...")
             s3_upload = s3.Uploader(self.env, self.logger)
-            s3_upload.trigger()
-            self.logger.info("S3 upload process completed.")
+            if s3_upload.trigger():
+                self.logger.error("Some objects failed to upload.")
+            else:
+                self.logger.info("S3 upload process completed.")
+                if not self.env.local_store:
+                    self.logger.info("Deleting local copy!")
+                    shutil.rmtree(self.clone_dir)
         else:
             self.logger.warning("No files found for S3 upload process.")

git2s3/s3.py

@@ -61,8 +61,13 @@ def upload_file(
         except (FileNotFoundError, BotoCoreError, ClientError) as error:
             raise exc.UploadError(error)
 
-    def trigger(self) -> None:
-        """Trigger to upload all file objects concurrently to S3."""
+    def trigger(self) -> bool:
+        """Trigger to upload all file objects concurrently to S3.
+
+        Returns:
+            bool:
+            Returns a boolean flag to indicate completion status.
+        """
         futures = {}
         with ThreadPoolExecutor(max_workers=os.cpu_count()) as executor:
             for root, dirs, files in os.walk(self.base_path):
@@ -74,10 +79,13 @@ def trigger(self) -> None:
                         self.upload_file, local_file_path, s3_file_path
                     )
                     futures[future] = s3_file_path
+        exception = False
         for future in as_completed(futures):
             if future.exception():
+                exception = True
                 self.logger.error(
                     "Thread processing '%s' received an exception: %s",
                     futures[future],
                     future.exception(),
                 )
+        return exception