skyplane-0.1.1

What's Changed

• AWS server provisioning by @parasj in #1
• GCP server provisioner by @parasj in #2
• Latency profiler by @parasj in #3
• Minor cleanup by @parasj in #4
• Bandwidth profiler by @parasj in #5
• Integrate benchmark to profile GCP Premium versus Standard network tiers by @parasj in #6
• Pareto evaluations of optimizer by @parasj in #7
• Gate cloud providers to not load GCP components if not evaluated by @parasj in #8
• Fix AWS and GCP key generation by @parasj in #9
• WIP: Dataplane by @ShishirPatil in #10
• Python dataplane test by @parasj in #11
• Build Python docker image by @parasj in #12
• Benchmark Python dataplane by @parasj in #13
• S3 interface by @parasj in #14
• Pipelining by @parasj in #15
• Initial null replicator (provision only + metadata API) by @parasj in #16
• [gateway] Add new API call to spawn and close server by @parasj in #17
• Add ChunkRequest API by @parasj in #18
• ReplicatorClient example sends ChunkRequests to source gateways by @parasj in #20
• Direct replication with randomly generated objects on source node by @parasj in #21
• Add new pytype workflow, fix some pytype issues by @parasj in #23
• Network graph with traceroute by @sarahwooders in #24
• Replicate across parallel instances by @parasj in #26
• 3-hop overlay works by @parasj in #28
• UDP benchmark by @parasj in #27
• Fix some provisioning issues with GCP and AWS by @parasj in #29
• Some minor fixes to build process by @parasj in #31
• Monitor chunk states using a new API call by @parasj in #30
• Use a single queue as opposed to per-worker queues by @samkumar in #33
• Standardize on gibibyte instead of gigabyte by @parasj in #34
• Provision a new Skylark VPC instead of using the default VPC by @parasj in #35
• Cleaner monitoring loop code w/ pandas by @parasj in #37
• Switch monitoring to netdata by @parasj in #39
• Connection pooling + tune batch size to eliminate stragglers by @parasj in #38
• Maintain a single persistent connection between Gateways by @parasj in #41
• Initial skylark CLI implementation by @parasj in #42
• GatewayReceivers should only close servers when they are done receiving all pending chunks by @ShishirPatil in #43
• Client web UI to display status of transfer by @parasj in #44
• Add past notebooks by @parasj in #46
• Benchmark triangles by @parasj in #45
• skylark replicate_random CLI tool by @parasj in #47
• Benchmark parallel connections by @parasj in #48
• Optionally enable BBR for benchmarks by @parasj in #49
• AWS limits CLI by @parasj in #50
• Dispatch ChunkRequests to source instances in parallel by @parasj in #51
• Send all chunk requests upfront to gateways instead of passing along in the critical path by @parasj in #52
• Benchmark multiple gateways by @parasj in #53
• Check AWS region is enabled before using it by @parasj in #54
• Fix miscellaneous bugs by @samkumar in #56
• Working GCP in gateway by @parasj in #58
• Add support for Microsoft Azure by @samkumar in #55
• Minor README by @ShishirPatil in #61
• Block till sufficient space is available by @parasj in #57
• Fix GCP firewall issue and increases banner timeout for SSH issue by @parasj in #64
• Cache public IPs by @parasj in #67
• [Client] Cancel pending chunk requests by closing gateways by @parasj in #70
• [CLI] Query available Azure SKUs by @parasj in #69
• [CLI] Initialize key cloud authentication details in a config file by @parasj in #65
• Use alternate VMs and disable some regions temporarily by @parasj in #76
• More robust throughput grid profiling by @parasj in #74
• Prompt to increase ulimit for files if it's too low, use fallocate to generate random files by @parasj in #77
• Use updated TCP window by @parasj in #78
• Basic optimizer fixes + optimizer CLI by @parasj in #81
• Fix solver to add connection limit and basic instance limit by @parasj in #82
• Integrate S3Interface with replicator by @sarahwooders in #72
• GCS Object Store Interface by @sarahwooders in #88
• GCSInterface integration with gateway + replicator by @sarahwooders in #91
• Solver instance limit and instance cost by @parasj in #87
• Generate instance-level plans from the solver by @parasj in #92
• Remove web dashboard by @parasj in #93
• Update gateway to run a ReplicationPlan graph by @parasj in #95
• Fix gateway replication by @parasj in #98
• Add GCS support to skylark cp CLI command by @sarahwooders in #97
• Azure Azure Blob Interface by @ShishirPatil in #99
• Migrate from loguru to print by @parasj in #104
• Transfer stall debugging by @parasj in #105
• Azure Gateway and replicator interface by @ShishirPatil in #100
• Exceptions parsing results should not stop throughput grid by @parasj in #110
• Update Dockerfile by @parasj in #111
• Ray autoscaler for Skylark control-plane by @ShishirPatil in #106
• Fix test_replicator_client by @parasj in #115
• CLI integration by @ShishirPatil in #114
• New solver updates for updated Pareto sweep by @parasj in #113
• Performance by @ShishirPatil in #116
• [DRAFT] Fix solver from bad state by @parasj in #119
• [DRAFT] Updated solver dispatch by @parasj in #118
• Work with arbitraty no of gateways by @ShishirPatil in #120
• Integrate skylark replicate-json CLI command with object stores by @sarahwooders in #117
• Process pool for uploading/downloading object store by @sarahwooders in #121
• Full skylark solver sweep by @parasj in #126
• Determine object store keys from listing bucket by @sarahwooders in #127
• Plotting azcopy, GCP Data Transfer, AWS DataSync by @ShishirPatil in #128
• Debug azure by @ShishirPatil in #129
• Load balance instances to avoid hotspots in a region by @parasj in #132
• End-to-end experiments (random + object stores) by @sarahwooders in #130
• Scale connections in the solver up to the instance max (64) by @parasj in #133
• Add updated plotting code for SIGCOMM histogram and pareto plots by @parasj in #136
• AWS DataSync client CLI by @parasj in #137
• Update plotting code to filter histograms by inter-cloud and intra-cloud by @parasj in #139
• Launch gateway with container optimized OS on GCP by @parasj in #153
• Use Amazon ECS optimized AMI which includes docker by @parasj in #152
• Tracing API to visualize upload + download progress on gateways by @parasj in #143
• Use a single resource group for all Azure resources by @samkumar in #146
• Fix strange straggler issues by @parasj in #158
• Flush status log queue to list periodically to avoid manager overhead by @parasj in #162
• Using Azure credentials from config by @ShishirPatil in #167
• Faster chunk status log requests via queue versus concurrent list by @parasj in #170
• Readme polishes by @antonzabreyko in #164
• Only initialize cloud keys for regions which are needed for a transfer by @parasj in #169
• Copy gateway logs to client machine by @parasj in #180
• Authentication from config by @ShishirPatil in #177
• Log transfer details to error logging dir by @parasj in #187
• Fix prompt ordering for "skylark init" by @samkumar in #186
• Updated plotting with new baseline results by @parasj in #154
• Added better error handling for disabled regions by @antonzabreyko in #190
• Skylark GCP CLI: ssh into gateway by @parasj in #194
• updated readme with netcat installation by @abiswal2001 in #189
• Fix gateway log download when transferring many chunks by @parasj in #202
• Issue 171: Fix on pr-limit bug by @abiswal2001 in #207
• Dead code cleanup by @parasj in #209
• New skylark init workflow with better config management by @parasj in #184
• Remove redundant size query by @antonzabreyko in #208
• Clarifying tweaks to documentation by @milesturin in #212
• Fix Issue #211 by @milesturin in #213
• Format code and add initial CONTRIBUTING doc by @parasj in #220
• Use TLS for inter-gateway traffic by @samkumar in #185
• Added region inference to GCS Interface by @abiswal2001 in #221
• Update required Python to 3.7 by @parasj in #223
• Partition batches with greedy algorithm by @parasj in #224
• Retry docker pull on error by @parasj in #218
• fixed race condition between keypair creation and replication by @milesturin in #225
• Automatically retry requests on HTTP error by @parasj in #219
• Replace SFTP creation with more robust client by @parasj in #230
• Fixed authentication error in Skylark Deprovision for uninitialized providers by @abiswal2001 in #232
• Remove unnecessary async logic in object store interface by @parasj in #233
• Handle signals via KeyboardException instead of atexit by @parasj in #227
• Add new state to queue downloads by @parasj in #234
• Copy objects between different source and destination prefixes by @parasj in #228
• Skylark CP Rework by @abiswal2001 in #226
• skylark init updates w/ explicit configuration by @parasj in #231
• Refactor dependencies for gateway by @parasj in #251
• Fix Azure integration, setup_bucket.py by @parasj in #254
• GCP region configuration by @antonzabreyko in #262
• [bug] Fix issue with cloud check for Azure by @parasj in #261
• Add Azure SSH command by @parasj in #260
• Enforce linter in Github Actions by @parasj in #264
• Update docker image used in skylark cp by @parasj in #265
• adding reuse gateways support to cp by @abiswal2001 in #274
• Improvements to skylark ls by @abiswal2001 in #263
• Documentation for Skyplane by @parasj in #250
• Fix Azure replication bugs by @parasj in #266
• cp rework 2 by @abiswal2001 in #284
• Handle VcpuLimitExceeded Exception in aws_cloud_provider.py by @ethanmehta in #256
• Latency grid for experiments CLI by @parasj in #282
• [experiment] Visualize one-hop cost tradeoff by @parasj in #288
• ablation experiment scripts by @abiswal2001 in #289
• [NSDI23] Single hop solver by @parasj in #291
• [experiment] Sweep parameters for cost-throughput tradeoff by @parasj in #287
• Remove old ablation logs by @parasj in #293
• Added GCP re-init condition if GCP config is missing. by @antonzabreyko in #286
• Fixed bug #237 skylark cp does not warn user if object does not exist by @ethanmehta in #248
• S3 boto3 migration (#198) by @milesturin in #255
• Fix support for whole bucket replication in skylark cp by @parasj in #302
• Fix bug with GCP init due to inferred_project_id by @parasj in #306
• Fix deprovisioning bug with Azure by @parasj in #297
• Replace oslo.concurrency with ilock by @parasj in #294
• Replace netcat dependency with socket check by @parasj in #295
• Create a new subnet in each availability zone for a region by @parasj in #283
• Document public Skylark API by @parasj in #292
• NSDI 2023 experiments by @parasj in #309
• Remove experiment code to move to experiments repository by @parasj in #307
• Fix #312: Provision fixed number of object store interfaces by @parasj in #313
• Fix issues with deprovisioning during replicate_helper by @parasj in #311
• Automated builds of documentation by @parasj in #315
• Improve UX of skylark cp tool with animations by @parasj in #314
• S3 Multipart Uploads & Downlods by @milesturin in #270
• Modify client, gateways, and chunks to support multipart uploads/downloads for large objects by @sarahwooders in #303
• Created basic CI framework by @milesturin in #326
• Exit replicator client upon any errors by @parasj in #324
• AWS firewall configuration via security groups by @ShishirPatil in #239
• Add CLI option to use BBR for transfers by @parasj in #331
• Clean up instance profiles in deprovision by @parasj in #334
• Fix #318 by passing project_id during setup to region config serialization by @parasj in #323
• Implement same region transfers for skylark cp by @parasj in #338
• Fix replicate random due to source instances bug by @parasj in #341
• Add performance logging for sockets per-transfer by @parasj in #342
• [draft] Remove duplicated replicator client code by @parasj in #348
• Refactor organization of CLI by @parasj in #347
• Fix skylark init bugs by @parasj in #350
• Skylark init caches Azure regions & SKU availability by @antonzabreyko in #339
• Fix GCP project configuration issues by @parasj in #352
• Automatically shut down gateways if not used by @parasj in #346
• Azure bug fix via forcing MSI credentials on gateways by @parasj in #353
• Skylark sync by @antonzabreyko in #356
• Rename occurrences of Skylark to Skyplane by @parasj in #305
• Set up Poetry based packaging by @parasj in #359
• Remove experiments, clean up code for mypy checking by @parasj in #362
• Add nightly release workflow by @parasj in #361
• Skyplane gateway data compression by @parasj in #363
• Log compression stats by @parasj in #364
• Auto-documentation support added by @abiswal2001 in #357
• Fix #365: cvxpy import error by @parasj in #367
• Fix #343: remove ILock dependency by @parasj in #370
• Fix #317: remove tqdm dependency by @parasj in #371
• Improve documentation by @parasj in #372
• Fix #366: explicit package discovery of skyplane package by @parasj in #369
• Fix #374: Remove tqdm dependency by @parasj in #377
• Fix #375 by conditionally importing Halo by @parasj in #376
• Fix #373: Relax dependency versions in pyproject.toml by @parasj in #378
• Replace halo with rich by @parasj in #379
• Remove termcolor, improve console output and add configuration CLI by @parasj in #380
• Print estimated egress cost and ask for confirmation by @parasj in #381
• Write CNAME on docs build by @parasj in #382
• [draft] README polishing by @parasj in #385
• CI tools by @milesturin in #384
• Use TLS for API Server by @samkumar in #349
• Debug GCP bug on main by @parasj in #401
• Fix wait_for issue by @parasj in #406
• Multipart uploads / downloads for GCP + service accounts for GCP by @sarahwooders in #360
• Delete existing service account keys by @parasj in #411
• Fix issue 368 by @antonzabreyko in #410
• Migrate internal CLI to new API by @parasj in #415
• Checksum at object stores by @parasj in #388
• Verify transfer at end of copy by @parasj in #407
• Checks ulimit in skyplane init, updates if necessary. Checks prlimits… #279 by @ethanmehta in #387
• Documentation Skeleton by @sarahwooders in #419
• Add new files for documentation by @sarahwooders in #420
• Run most CI checks on push by @parasj in #421
• [ux] catch top level exceptions with pretty printing #397 by @ethanmehta in #414
• Improve performance of gateway by improving performance of GatewayDaemonAPI SSL by @parasj in #418
• Implement end-to-end encryption using NaCl secretbox by @samkumar in #416
• Add configuration options for key transfer parameters by @parasj in #425
• [GCP] skyplane VPC and firewall by @ShishirPatil in #426
• Deprecate local transfers by @parasj in #430
• Dev/shishir/gcp fix by @ShishirPatil in #432
• Fix pytype errors and minor bugs in GCP firewall by @parasj in #433
• Catch error in ulimit check by @parasj in #436
• Remove questionary dependency by @parasj in #434
• Debugging docs by @sarahwooders in #437
• Reduce SKU query parallelism by @parasj in #431
• Improve path resolution logic and add new unit test by @parasj in #423
• [Doc] Firewalls by @ShishirPatil in #440
• Transfer validation documentation by @parasj in #441
• Remove create_bucket option from ObjStoreInterface by @parasj in #443
• Partial application of minor Azure fixes by @parasj in #444
• Fix Azure transfers with additional permissions by @parasj in #435
• Autodocumentation by @parasj in #442
• Install Skyplane before generating docs by @parasj in #445
• AWS integration tests by @parasj in #446
• Add GCP to GCP integration test by @parasj in #447
• Debug GCP service account key issue by @parasj in #449
• Update integration test with new service account upon deprovision by @parasj in #450
• Documentation improvements by @parasj in #448
• GCP integration tests stability by @parasj in #451
• Correctly deprovision instances by @parasj in #452
• Push Docker builds to from ghcr.io to AWS ECR by @parasj in #453
• [draft] Emit structured tags by @parasj in #454
• Tie nightly PyPI release to GATEWAY_VERSION by @parasj in #455
• Bind PyPI release to a specific docker image by @parasj in #457
• Add profiles to package by @parasj in #458
• Remove ulimit check by @parasj in #459
• Bump version to 0.0.1a1 and release package by @parasj in #460

New Contributors

• @ShishirPatil made their first contribution in #10
• @samkumar made their first contribution in #33
• @antonzabreyko made their first contribution in #164
• @abiswal2001 made their first contribution in #189
• @milesturin made their first contribution in #212
• @ethanmehta made their first contribution in #256

Full Changelog: https://github.com/skyplane-project/skyplane/commits/0.1.1