01_standalone_examples/migrate-or-clone-repo/Migrate or Clone Local Repo to AWS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4e81663-80e6-4d27-b5d8-788660b20453",
   "metadata": {},
   "source": [
    "# Migrate or clone a local lakeFS repository to AWS\n",
    "\n",
    "#### Use this notebook if you want to migrate/clone a local source repository to a target repository on AWS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70041e78-71fc-4222-84eb-adb47f2fec33",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "#### 1. Source and a target lakeFS environments (you can [deploy one independently](https://docs.lakefs.io/deploy/) or use the hosted solution [lakeFS Cloud](https://lakefs.cloud))\n",
    "#### 2. Source repository\n",
    "#### 3. S3 Bucket for target repository but don't create target repository ahead of time (you will create a target repo in this notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a39a3a-c7d8-4f56-a56e-b7f67de561ca",
   "metadata": {},
   "source": [
    "## Setup Task: Import required Python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8fb6261-08b1-4daa-8866-e417f3b1f302",
   "metadata": {},
   "outputs": [],
   "source": [
    "%xmode Minimal\n",
    "import lakefs\n",
    "from lakefs.client import Client\n",
    "import lakefs_sdk\n",
    "from lakefs_sdk.client import LakeFSClient\n",
    "import random\n",
    "import os\n",
    "import datetime\n",
    "from awscliv2.api import AWSAPI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95cf9ca3-5044-4dc1-90ac-09eb6b1b869c",
   "metadata": {},
   "source": [
    "## Setup Task: Change your lakeFS credentials for the local source lakeFS environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5483a303-964d-4555-b7df-680adc52d8b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sourceLakefsEndPoint = '<Source lakeFS Endpoint URL>' # e.g. 'http://host.docker.internal:8000' if you are running lakeFS in a separate Docker container on your local machine\n",
    "sourceLakefsAccessKey = '<Source lakeFS Access Key>'\n",
    "sourceLakefsSecretKey = '<Source lakeFS Secret Key>'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a51da4f5-2943-4a00-a412-5a1955317a6f",
   "metadata": {},
   "source": [
    "## Setup Task: Change your lakeFS credentials for the target lakeFS environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23f11814-70b4-4a19-adf4-a62306b1c5c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "targetLakefsEndPoint = '<Target lakeFS Endpoint URL>' # e.g. 'https://username.aws_region_name.lakefscloud.io'\n",
    "targetLakefsAccessKey = '<Target lakeFS Access Key>'\n",
    "targetLakefsSecretKey = '<Target lakeFS Secret Key>'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21ae19fa-acd8-4aa0-af93-cdd9bed56487",
   "metadata": {},
   "source": [
    "## Setup Task: Change lakeFS repo names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4518a8fc-e5e9-42ec-a0b7-5a000833029c",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_repo_name = \"source-repo\"\n",
    "target_repo_name = \"target-repo\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c7c9735-e8f5-4122-ba9e-e2f9dd44bd22",
   "metadata": {},
   "source": [
    "## Setup Task: Change main/production branch name for the source repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94991d59-4bc9-44cd-96d1-f6bc8c29becf",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_main_branch = \"main\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86ee9f0-98d4-406c-9ad8-200a7f97c05d",
   "metadata": {},
   "source": [
    "## Setup Task: Change source and target storage namespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e05f472-271b-465c-ae68-6f888869c1ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_storage_namespace = 'local://localSourceFolder' # storage namespace for the local source repo\n",
    "target_storage_namespace = 's3://myTargetBucket' # S3 bucket or any prefix/folder within the bucket e.g. 's3://myTargetBucket/<prefix>'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "514bb5ff-1ee6-4310-a0fd-6ef46a66b06e",
   "metadata": {},
   "source": [
    "## Setup Task: Change AWS credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e48ec9a2-4a7d-416d-b83e-e7daead9c17e",
   "metadata": {},
   "outputs": [],
   "source": [
    "aws_region = '<AWS region name>' # e.g. us-east-1\n",
    "aws_access_key_id = 'aaaaaaaaaaaaa'\n",
    "aws_secret_access_key = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "457da24d-7644-41fe-bb00-9d3e4c363bb1",
   "metadata": {},
   "source": [
    "## Setup Task: Set AWS credentials for the default profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57368f0d-09cd-47df-b7bb-ed4008bca7c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "aws_api = AWSAPI()\n",
    "\n",
    "aws_api.set_credentials(\"default\", aws_access_key_id, aws_secret_access_key, \"\", aws_region)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a55daa-fe3c-47ed-b353-e7cd26dc5f57",
   "metadata": {},
   "source": [
    "## Setup Task: Create lakeFS Python client for source lakeFS environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25ceb3de-d811-41fa-b117-ce6e15de293a",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not 'source_lakefs_client' in locals():\n",
    "    source_lakefs_client = Client(\n",
    "        host=sourceLakefsEndPoint,\n",
    "        username=sourceLakefsAccessKey,\n",
    "        password=sourceLakefsSecretKey,\n",
    "    )\n",
    "    \n",
    "    configuration = lakefs_sdk.Configuration(\n",
    "        host=sourceLakefsEndPoint,\n",
    "        username=sourceLakefsAccessKey,\n",
    "        password=sourceLakefsSecretKey,\n",
    "    )\n",
    "    source_lakefs_sdk_client = LakeFSClient(configuration)\n",
    "    \n",
    "    print(\"Verifying lakeFS credentials…\")\n",
    "    try:\n",
    "        v=source_lakefs_client.version\n",
    "        sourceRepo = lakefs.Repository(source_repo_name, client=source_lakefs_client)\n",
    "    except:\n",
    "        print(\"🛑 failed to get lakeFS version\")\n",
    "    else:\n",
    "        print(f\"…✅lakeFS credentials verified\\n\\nℹ️lakeFS version {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1e05564-131f-4a90-81ca-16dfa1fc2ec4",
   "metadata": {},
   "source": [
    "## Setup Task: Create lakeFS Python client for target lakeFS environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9ffe9b2-5c84-4637-bde6-5ebf40e49d10",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not 'target_lakefs_client' in locals():\n",
    "    target_lakefs_client = Client(\n",
    "        host=targetLakefsEndPoint,\n",
    "        username=targetLakefsAccessKey,\n",
    "        password=targetLakefsSecretKey,\n",
    "    )\n",
    "    \n",
    "    configuration = lakefs_sdk.Configuration(\n",
    "        host=targetLakefsEndPoint,\n",
    "        username=targetLakefsAccessKey,\n",
    "        password=targetLakefsSecretKey,\n",
    "    )\n",
    "    target_lakefs_sdk_client = LakeFSClient(configuration)\n",
    "    \n",
    "    print(\"Verifying lakeFS credentials…\")\n",
    "    try:\n",
    "        v=target_lakefs_client.version\n",
    "    except:\n",
    "        print(\"🛑 failed to get lakeFS version\")\n",
    "    else:\n",
    "        print(f\"…✅lakeFS credentials verified\\n\\nℹ️lakeFS version {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8277d49c-570d-4ade-a976-952ce92e4451",
   "metadata": {},
   "source": [
    "# Step 1 - Commit Changes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35130b5-ef88-44c4-a90b-3969784c3259",
   "metadata": {},
   "source": [
    "## IMPORTANT: Uncommitted data is not migrated over so check uncommitted data (this might take time if you have many branches in the source repo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37ff37f6-0915-42aa-b8f7-b89bff042240",
   "metadata": {},
   "outputs": [],
   "source": [
    "for branchList in sourceRepo.branches():\n",
    "    for diff in sourceRepo.branch(branchList.id).uncommitted():\n",
    "        print('Branch with uncommitted data: ' + branchList.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5129aa0d-9f90-4f38-a2a9-f0111a15bcc1",
   "metadata": {},
   "source": [
    "## OPTIONAL: Commit any uncommitted data in your source repo (this might take time if you have many branches in the source repo)\n",
    "#### If you want, you can manually commit the changes for the branches listed above from the lakeFS UI\n",
    "\n",
    "#### You can run previous command again after finishing this to verify that there are not any uncommitted data any more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3059fc2d-596b-44c2-8401-2b05da5c517b",
   "metadata": {},
   "outputs": [],
   "source": [
    "for branchList in sourceRepo.branches():\n",
    "    for diff in sourceRepo.branch(branchList.id).uncommitted():\n",
    "        ref = sourceRepo.branch(branchList.id).commit(message='Committed changes during the migration of the repository')\n",
    "        print(ref.get_commit())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89dcf103-090d-43a2-bb60-080a0dbe828d",
   "metadata": {},
   "source": [
    "# Step 2 - Dump Metadata of Source Repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9bb3f60-8080-4b3c-8fa5-d8f5e2508f7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_lakefs_sdk_client.internal_api.dump_refs(source_repo_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e789b515-2333-4d45-879c-130afbd8ef85",
   "metadata": {},
   "source": [
    "# Step 3 - Copy Data from Source to Target"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab164cf-c20d-47d9-87c1-d50d544e9446",
   "metadata": {},
   "source": [
    "#### You can directly copy data from local storage to target storage on your own\n",
    "#### or you can run following printed command on your local machine to copy data from local Docker container to local machine first\n",
    "#### (change the Docker container name for lakeFS and go to the folder where you cloned lakefs-samples Git repo before running the command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e05e8a7b-76e4-42b3-8357-bb533d97b26f",
   "metadata": {},
   "outputs": [],
   "source": [
    "lakefs_docker_container_name = 'lakefs'\n",
    "print('docker cp ' + lakefs_docker_container_name + ':/home/lakefs/lakefs/data/block/' + source_storage_namespace.split('://')[1] + \\\n",
    "      '/ lakeFS-samples/01_standalone_examples/migrate-or-clone-repo/localDownloadedSourceFolder/')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2965d95-9af4-4959-85a1-c6231b07de70",
   "metadata": {},
   "source": [
    "#### Now copy data from your local machine to target storage on S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7bf575-cc71-4b57-9fcf-c09193989492",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3SyncCommand = 'aws s3 sync ./localDownloadedSourceFolder/ ' + target_storage_namespace\n",
    "print(s3SyncCommand)\n",
    "\n",
    "! $s3SyncCommand"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20314a68-533e-45f3-bd09-ac6be17a34cc",
   "metadata": {},
   "source": [
    "## Step 4 - Create Target Bare Repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29e91a48-6601-45d4-a8aa-6c5de808d208",
   "metadata": {},
   "outputs": [],
   "source": [
    "lakefs.Repository(target_repo_name, client=target_lakefs_client).create(storage_namespace=f\"{target_storage_namespace}\", default_branch=source_main_branch, bare=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95570b12-b39d-41d7-852c-f09dc4b05bdf",
   "metadata": {},
   "source": [
    "## Step 5 - Restore Metadata to Target Repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cef6e90a-fd2d-42ff-b1ae-053a619ec560",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_lakefs_sdk_client.internal_api.restore_refs(target_repo_name, source_lakefs_sdk_client.internal_api.dump_refs(source_repo_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b1304e-e4db-4d08-9662-c8d4547c3269",
   "metadata": {},
   "source": [
    "## More Questions?\n",
    "\n",
    "###### Join the lakeFS Slack group - https://lakefs.io/slack"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}