Skip to content

Commit

Permalink
final fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
ahmeda14960 committed Oct 24, 2024
1 parent 4733f3b commit e82eec2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/levanter/data/text.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ async def current_len(self) -> Optional[int]:
async def get_batch(self, indices: Sequence[int]) -> Sequence[T_co]:
# Use self.wait_until_len_at_least to ensure we have enough data for the batch.
max_index = max(indices)
ds_len = await self.wait_until_len_at_least(max_index + 1)
ds_len = await self.dataset.wait_until_len_at_least(max_index + 1)

# Determine the epoch based on the largest index
epoch = max_index // ds_len
Expand Down

0 comments on commit e82eec2

Please sign in to comment.