Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set associative backup #30

Open
wants to merge 96 commits into
base: gem5-develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
7260f43
mem-DRAM-cache-with-stdlib
aakahlow Mar 23, 2023
3a5974f
mem: small fixes to compile gem5 after rebase
aakahlow Mar 24, 2023
1e364f0
mem: Add Octopi cache test
aakahlow Mar 24, 2023
5040bdb
mem: small fixes
aakahlow Mar 24, 2023
f13d445
mem: dcache with ports -- working version
aakahlow Mar 25, 2023
4ab8cdf
configs: Use DiskImageWorkload and fix path
powerjg Mar 27, 2023
6396d45
stdlib: fix cache line size error
aakahlow Mar 27, 2023
53d8fd7
stdlib: temp change
aakahlow Mar 27, 2023
6451a98
Merge branch 'stdlib-dram-cache' of https://github.com/darchr/dramCac…
aakahlow Mar 27, 2023
0ea5160
configs: HACK\! enable checkpointing in octotpi
powerjg Mar 27, 2023
6de38a0
stdlib: Sync changes
aakahlow Mar 27, 2023
2ef6728
Merge branch 'stdlib-dram-cache' of https://github.com/darchr/dramCac…
aakahlow Mar 27, 2023
9f9e7d8
mem: cache warmup exit handling
aakahlow Mar 28, 2023
2b5bc05
misc: Add checkpoint script
aakahlow Mar 28, 2023
ff1c8c0
mem: some minor updates
mbabaie Mar 23, 2023
80951fa
mem: just the GAPBS and NPB scripts are updated
mbabaie Mar 23, 2023
5130376
mem: latest version, modified for write age be ignored for cases not …
mbabaie Mar 24, 2023
a5a2c46
mem: updated stats
mbabaie Mar 27, 2023
6948494
mem: just updated some stats and params
mbabaie Mar 29, 2023
9cfb24d
mem: more updates on stats
mbabaie Mar 29, 2023
42e2de8
stdlib: checkpoint script
aakahlow Mar 29, 2023
d3802b0
mem: syncing changes
aakahlow Apr 2, 2023
e7c4ff9
mem: update scripts
aakahlow Apr 7, 2023
bebd0ec
misc: documentation
aakahlow Apr 7, 2023
e05c42f
stdlib: add missing local change
aakahlow Apr 8, 2023
c48f8be
misc: doc update
aakahlow Apr 8, 2023
7295178
mem-ruby: Fix warmup with backing store
powerjg Apr 8, 2023
5b5a859
arch-riscv: Copy miscregs on takeOverFrom
powerjg Apr 9, 2023
9f6b6a9
ckpt restore script for gapbs
aakahlow Apr 9, 2023
d12315c
o3: Add looppoint stuff in this branch
aakahlow Apr 14, 2023
d3f2b3c
Update restore script
aakahlow Apr 14, 2023
6bc2f76
small changes
aakahlow Apr 14, 2023
7154f0c
syncing changes
aakahlow Apr 17, 2023
bf9dddb
small changes
aakahlow Apr 17, 2023
ce13de0
mem: removed old files
mbabaie Mar 29, 2023
76dee96
mem: created TDRAM_32
mbabaie Mar 30, 2023
0cd74f3
mem: modified scripts traff gen
mbabaie Mar 30, 2023
03267a2
mem: added plot data
mbabaie Apr 3, 2023
652ed8d
mem: added some test cases
mbabaie Apr 5, 2023
54911df
mem: latest plots
mbabaie Apr 5, 2023
75ff74e
mem:latest version
mbabaie Apr 17, 2023
46496f7
mem: updated scripts and merged with latest version of TDRAM
mbabaie Apr 17, 2023
5e77193
mem: updated old scripts
mbabaie Apr 18, 2023
0873402
mem: updated old scripts
mbabaie Apr 18, 2023
d369d71
python: Allow SimObjectVector to match proxy
powerjg Apr 18, 2023
9bd3570
mem-ruby: Allow multiple backing store ranges
powerjg Apr 18, 2023
d5e7c00
configs: Updates to system to work with more memory
powerjg Apr 18, 2023
d193060
mem: updated scripts
mbabaie Apr 19, 2023
2d9595b
mem: updated scripts
mbabaie Apr 19, 2023
b276dff
mem: fixed the exitSimLoop error
mbabaie Apr 20, 2023
43a04bf
cpu-o3: Update looppoint analysis
powerjg Apr 20, 2023
7018c5c
misc: Add region info for workloads
powerjg Apr 20, 2023
d012507
just some updates on the scripts
mbabaie Apr 20, 2023
517fa03
updated scripts, used for chkpts and restores
mbabaie Apr 21, 2023
1f877d1
configs: Add restore both script and update info
powerjg Apr 21, 2023
b54e03d
cpu-o3: Update looppoint probe to keep order
powerjg Apr 21, 2023
1c3f005
Merge branch 'stdlib-dram-cache' of https://github.com/darchr/dramCac…
mbabaie Apr 21, 2023
27ba5eb
cpu-o3: Fix off by one in looppoint analysis
powerjg Apr 21, 2023
c59c442
Revert "cpu-o3: Fix off by one in looppoint analysis"
powerjg Apr 21, 2023
06f3621
cpu: Update looppoint pc-count hash function
powerjg Apr 22, 2023
9694b70
configs: add workload info
powerjg Apr 22, 2023
64e0306
configs: update info with 24 hour point
powerjg Apr 22, 2023
618f59f
Merge branch 'stdlib-dram-cache' of https://github.com/darchr/dramCac…
mbabaie Apr 24, 2023
6e6a10d
fixed some status for Oracle
mbabaie Apr 24, 2023
41f543a
configs: Update info with D/25 classes
powerjg Apr 24, 2023
fc7c9c8
configs: Update restore both script
powerjg Apr 24, 2023
f9824c1
revised version for combined dirmap and set-asso, only tested for dirmap
mbabaie May 22, 2023
30eb555
first version tested with traff gen and tc in GAP
mbabaie May 31, 2023
23a5f56
mem: added serialize and unserialize for chkpt and rstr
mbabaie May 31, 2023
c001ad6
latest version of set-associative
mbabaie Jun 6, 2023
5de6343
mem: added data plot for set-assoc
mbabaie Jun 6, 2023
29881e7
mem: added data plot for set-assoc
mbabaie Jun 6, 2023
d37edc1
fixed unserialization
mbabaie Jun 7, 2023
1ecfd31
new set-assoc data
mbabaie Jun 7, 2023
51f3eb0
mem: latest version
mbabaie Jun 12, 2023
94971bd
fixing tag issue in unserialization
mbabaie Jun 13, 2023
196a710
data plots
mbabaie Jun 13, 2023
7983b55
data plots
mbabaie Jun 13, 2023
4490f78
data plots
mbabaie Jun 13, 2023
b615162
updated plots
mbabaie Jun 13, 2023
cc7bf0c
misc: Google traces files
aakahlow Jul 13, 2023
382ac09
misc: gtraces config
aakahlow Jul 13, 2023
df9af97
data plots for comparing running for longer hours
mbabaie Jul 14, 2023
1a7a87c
data plots for comparing running for longer hours
mbabaie Jul 14, 2023
5a62c5c
mem: updated google scripts
mbabaie Jul 15, 2023
eef1be6
mem: updated google scripts
mbabaie Jul 15, 2023
fb7cef1
small changes in scripts for chkpt and pc analysis
mbabaie Jul 15, 2023
da78e9c
small changes in scripts for chkpt and pc analysis
mbabaie Jul 15, 2023
7052946
updated scripts with new pc
mbabaie Jul 16, 2023
4df4590
updated scripts with new pc
mbabaie Jul 16, 2023
399b250
updated scripts
mbabaie Jul 16, 2023
c4f2c6b
backup
mbabaie Jul 27, 2023
1baf28a
parent 41af1e744aaf2a2dd291d32ad1ec1b672b8822b9 author mbabaie <mbaba…
mbabaie Jul 17, 2023
e7c04a3
just some comments removed
mbabaie Sep 2, 2023
ae11005
tiny changes in the script
Sep 13, 2023
7b5b705
tested archs
mbabaie Oct 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions Methodology.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Documentation on the methodology work for DRAM cache experiments

This is not a very detailed or formal document, but tries to at least point to
the relevant links or paths and important notes.

- DRAM cache component is part of the stdlib in this repo ("src/python/gem5/components/memory/dcache.py").
- Octopi-cache is a three level MESI cache hiearchy


## gem5 scripts used to take checkpoints

- **NPB D Class:** Octopi-cache/riscv-2channel-1ccd-checkpoint-timing.py

- **NPB C Class:** ctopi-cache/riscv-2channel-1ccd-checkpoint-timing.py

- **GAPBS (2^25 vertices):** Octopi-cache/riscv-2channel-1ccd-checkpoint-timing-gapbs.py


## Bash scripts used (will show the directory where the checkpoints and results are)

- **NPB:** `npb_checkpoint.sh` and `npb_checkpoint_c_class.sh`
- **GAPBS:** `gapbs_checkpoint.sh`

The gem5 directory I have been using is here: `/home/aakahlow/Dcache_Src/dramCacheController`.
D Class and GAPBS checkpoints will go there. The C class checkpoints will be stored in /projects (exact path in the npb_checkpoint_c_class.sh script).

## Looppoint stuff

- The branch with O3 loop point analysis stuff: https://github.com/darchr/gem5/commits/looppointAnalysis-Ayaz
- example.py script in the above branch is a modified version of Zhantong's example script to use looppoint analysis probe and print most recent PC + count.

162 changes: 162 additions & 0 deletions Octopi-cache/components/Octopi.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
from gem5.components.cachehierarchies.ruby.abstract_ruby_cache_hierarchy import AbstractRubyCacheHierarchy
from gem5.components.cachehierarchies.abstract_three_level_cache_hierarchy import (
AbstractThreeLevelCacheHierarchy,
)
from gem5.coherence_protocol import CoherenceProtocol
from gem5.isas import ISA
from gem5.components.boards.abstract_board import AbstractBoard
from gem5.utils.requires import requires

from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.directory import Directory
from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.dma_controller import DMAController

from m5.objects import RubySystem, DMASequencer, RubyPortProxy, SimpleMemory, AddrRange

from .core_complex import CoreComplex
from .octopi_network import OctopiNetwork
from .ruby_network_components import RubyNetworkComponent, RubyRouter, RubyExtLink, RubyIntLink

# CoreComplex sub-systems own the L1, L2, L3 controllers
# OctopiCache owns the directory controllers
# RubySystem owns the DMA Controllers
class OctopiCache(AbstractRubyCacheHierarchy, AbstractThreeLevelCacheHierarchy):
def __init__(
self,
l1i_size: str,
l1i_assoc: int,
l1d_size: str,
l1d_assoc: int,
l2_size: str,
l2_assoc: int,
l3_size: str,
l3_assoc: int,
num_core_complexes: int,
is_fullsystem: bool
):
AbstractRubyCacheHierarchy.__init__(self=self)
AbstractThreeLevelCacheHierarchy.__init__(
self=self,
l1i_size=l1i_size,
l1i_assoc=l1i_assoc,
l1d_size=l1d_size,
l1d_assoc=l1d_assoc,
l2_size=l2_size,
l2_assoc=l2_assoc,
l3_size=l3_size,
l3_assoc=l3_assoc,
)

self._directory_controllers = []
self._dma_controllers = []
self._io_controllers = []
self._core_complexes = []
self._num_core_complexes = num_core_complexes
self._is_fullsystem = is_fullsystem

def incorporate_cache(self, board: AbstractBoard) -> None:

requires(
coherence_protocol_required=CoherenceProtocol.MESI_THREE_LEVEL
)

cache_line_size = board.get_cache_line_size()

self.ruby_system = RubySystem()
# MESI_Three_Level needs 3 virtual networks
self.ruby_system.number_of_virtual_networks = 3
self.ruby_system.network = OctopiNetwork(self.ruby_system)
self.ruby_system.access_backing_store = True

# Get the first range and the range part.
addr_range_0 = board.get_mem_ports()[0][0]
self.ruby_system.phys_mem = SimpleMemory(
range=AddrRange(start=addr_range_0.start,
end=addr_range_0.end),
in_addr_map=False)

# Setting up the core complex
all_cores = board.get_processor().get_cores()
num_cores_per_core_complex = len(all_cores) // self._num_core_complexes

self.core_complexes = [CoreComplex(
board = board,
cores = all_cores[core_complex_idx*num_cores_per_core_complex:(core_complex_idx + 1) * num_cores_per_core_complex],
ruby_system = self.ruby_system,
l1i_size = self._l1i_size,
l1i_assoc = self._l1i_assoc,
l1d_size = self._l1d_size,
l1d_assoc = self._l1d_assoc,
l2_size = self._l2_size,
l2_assoc = self._l2_assoc,
l3_size = self._l3_size,
l3_assoc = self._l3_assoc,
) for core_complex_idx in range(self._num_core_complexes)]

self.ruby_system.network.incorporate_ccds(self.core_complexes)

self._create_directory_controllers(board)
self._create_dma_controllers(board, self.ruby_system)

self.ruby_system.num_of_sequencers = len(all_cores) + len(self._dma_controllers) + len(self._io_controllers)
# SimpleNetwork requires .int_links and .routers to exist
# if we want to call SimpleNetwork.setup_buffers()
self.ruby_system.network.int_links = self.ruby_system.network._int_links
self.ruby_system.network.ext_links = self.ruby_system.network._ext_links
self.ruby_system.network.routers = self.ruby_system.network._routers
self.ruby_system.network.setup_buffers()

# Set up a proxy port for the system_port. Used for load binaries and
# other functional-only things.
self.ruby_system.sys_port_proxy = RubyPortProxy()
board.connect_system_port(self.ruby_system.sys_port_proxy.in_ports)

def _create_directory_controllers(self, board):
# Adding controllers
self.directory_controllers = [Directory(
self.ruby_system.network, board.get_cache_line_size(), addr_range, mem_port
) for addr_range, mem_port in board.get_mem_ports()
]
for ctrl in self.directory_controllers:
ctrl.ruby_system = self.ruby_system
# Adding controller routers
self.directory_controller_routers = [RubyRouter(self.ruby_system.network) for _ in range(len(self.directory_controllers))]
for router in self.directory_controller_routers:
self.ruby_system.network._add_router(router)
# Adding an external link for each controller and its router
self.directory_controller_ext_links = [RubyExtLink(ext_node=dir_ctrl, int_node=dir_router) for dir_ctrl, dir_router in zip(self.directory_controllers, self.directory_controller_routers)]
for ext_link in self.directory_controller_ext_links:
self.ruby_system.network._add_ext_link(ext_link)
_directory_controller_int_links = []
for router in self.directory_controller_routers:
int_link_1, int_link_2 = RubyIntLink.create_bidirectional_links(router, self.ruby_system.network.cross_ccd_router)
_directory_controller_int_links.extend([int_link_1, int_link_2])
self.ruby_system.network._add_int_link(int_link_1)
self.ruby_system.network._add_int_link(int_link_2)
self.directory_controller_int_links = _directory_controller_int_links

def _create_dma_controllers(self, board, ruby_system):
# IOController for full system simulation
if self._is_fullsystem:
self.io_sequencer = DMASequencer(version=0, ruby_system=self.ruby_system)
self.io_sequencer.in_ports = board.get_mem_side_coherent_io_port()
self.ruby_system.io_controller = DMAController(
dma_sequencer=self.io_sequencer,
ruby_system=self.ruby_system
)
self._io_controllers.append(self.ruby_system.io_controller)
self.io_controller_router = RubyRouter(self.ruby_system.network)
self.ruby_system.network._add_router(self.io_controller_router)
self.io_controller_ext_link = RubyExtLink(ext_node=self._io_controllers[0], int_node=self.io_controller_router)
self.ruby_system.network._add_ext_link(self.io_controller_ext_link)
self.io_controller_int_links = RubyIntLink.create_bidirectional_links(self.io_controller_router, self.ruby_system.network.cross_ccd_router)
self.ruby_system.network._add_int_link(self.io_controller_int_links[0])
self.ruby_system.network._add_int_link(self.io_controller_int_links[1])

self._dma_controllers = []
if board.has_dma_ports():
for i, port in enumerate(dma_ports):
ctrl = DMAController(self.ruby_system.network, cache_line_size)
ctrl.dma_sequencer = DMASequencer(version=i+1, in_ports=port)
self._dma_controllers.append(ctrl)
ctrl.ruby_system = self.ruby_system
ruby_system.dma_controllers = self._dma_controllers
Empty file.
192 changes: 192 additions & 0 deletions Octopi-cache/components/core_complex.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
from typing import List, Tuple

from gem5.isas import ISA
from gem5.components.boards.abstract_board import AbstractBoard
from gem5.components.processors.abstract_core import AbstractCore
from gem5.components.cachehierarchies.abstract_three_level_cache_hierarchy import (
AbstractThreeLevelCacheHierarchy,
)
from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.l1_cache import L1Cache
from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.l2_cache import L2Cache
from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.l3_cache import L3Cache
from gem5.components.cachehierarchies.ruby.caches.mesi_three_level.directory import Directory

from m5.objects import SubSystem, L2Cache_Controller, AddrRange, RubySequencer, Switch, SimpleIntLink, SimpleExtLink, SubSystem, SimObject

from .ruby_network_components import RubyRouter, RubyExtLink, RubyIntLink, RubyNetworkComponent

class CoreComplex(SubSystem, RubyNetworkComponent):
_core_id = 0
_core_complex_id = 0

@classmethod
def _get_core_id(cls):
cls._core_id += 1
return cls._core_id - 1

@classmethod
def _get_core_complex_id(cls):
cls._core_complex_id += 1
return cls._core_complex_id - 1

def __init__(
self,
board: AbstractBoard,
cores: List[AbstractCore],
ruby_system,
l1i_size: str,
l1i_assoc: int,
l1d_size: str,
l1d_assoc: int,
l2_size: str,
l2_assoc: int,
l3_size: str,
l3_assoc: int,
):
SubSystem.__init__(self=self)
RubyNetworkComponent.__init__(self=self)

self._l1i_size = l1i_size
self._l1i_assoc = l1i_assoc
self._l1d_size = l1d_size
self._l1d_assoc = l1d_assoc
self._l2_size = l2_size
self._l2_assoc = l2_assoc
self._l3_size = l3_size
self._l3_assoc = l3_assoc

self._board = board
self._cores = cores
self._ruby_system = ruby_system
self._cache_line_size = 64

self._directory_controllers = []

self._core_complex_id = self._get_core_complex_id()
self.main_router = RubyRouter(self._ruby_system) # this will be connect to component outside the core complex
self._add_router(self.main_router)
self._create_core_complex()

def get_main_router(self):
return self.main_router

def _create_core_complex(self):
# Create L1 caches, L2 cache, and corresponding controllers per core
self.core_clusters = [self._create_core_cluster(core) for core in self._cores]
# Create L3 cache and its corresponding controller
self._create_shared_cache()
# Setting up one router and one external link per controller
self._create_external_links()
# Setting up L1/L2 links, L2/main links, L3/main link
self._create_internal_links()

def _create_core_cluster(self, core: AbstractCore):
cluster = SubSystem()
core_id = self._get_core_id()

cluster.l1_cache = L1Cache(
l1i_size = self._l1i_size,
l1i_assoc = self._l1i_assoc,
l1d_size = self._l1d_size,
l1d_assoc = self._l1d_assoc,
network = self._ruby_system.network,
core = core,
cache_line_size = self._cache_line_size,
target_isa = self._board.processor.get_isa(),
clk_domain = self._board.get_clock_domain(),
)
cluster.l1_cache.sequencer = RubySequencer(
version = core_id,
dcache = cluster.l1_cache.Dcache,
clk_domain = cluster.l1_cache.clk_domain
)

if self._board.has_io_bus():
cluster.l1_cache.sequencer.connectIOPorts(self._board.get_io_bus())
cluster.l1_cache.ruby_system = self._ruby_system
core.connect_icache(cluster.l1_cache.sequencer.in_ports)
core.connect_dcache(cluster.l1_cache.sequencer.in_ports)
core.connect_walker_ports(
cluster.l1_cache.sequencer.in_ports, cluster.l1_cache.sequencer.in_ports
)
if self._board.get_processor().get_isa() == ISA.X86:
core.connect_interrupt(
cluster.l1_cache.sequencer.interrupt_out_port,
cluster.l1_cache.sequencer.in_ports
)
else:
core.connect_interrupt()

cluster.l2_cache = L2Cache(
l2_size=self._l2_size,
l2_assoc=self._l2_assoc,
network=self._ruby_system.network,
core=core,
num_l3Caches=1, # each core complex has 1 slice of L3 Cache
cache_line_size=self._cache_line_size,
cluster_id=self._core_complex_id,
target_isa=self._board.processor.get_isa(),
clk_domain=self._board.get_clock_domain(),
)
cluster.l2_cache.ruby_system = self._ruby_system
# L0Cache in the ruby backend is l1 cache in stdlib
# L1Cache in the ruby backend is l2 cache in stdlib
cluster.l2_cache.bufferFromL0 = cluster.l1_cache.bufferToL1
cluster.l2_cache.bufferToL0 = cluster.l1_cache.bufferFromL1

return cluster

def _create_shared_cache(self):
self.l3_cache = L3Cache(
l3_size=self._l3_size,
l3_assoc=self._l3_assoc,
network=self._ruby_system.network,
num_l3Caches=1,
cache_line_size=self._cache_line_size,
cluster_id=self._core_complex_id,
)
self.l3_cache.ruby_system = self._ruby_system

# This is where all routers and links are created
def _create_external_links(self):
# create a router per cache controller
# - there is one L3 per ccd
self.l3_router = RubyRouter(self._ruby_system)
self._add_router(self.l3_router)
# - there is one L1 and one L2 per cluster
for cluster in self.core_clusters:
cluster.l1_router = RubyRouter(self._ruby_system)
self._add_router(cluster.l1_router)
cluster.l2_router = RubyRouter(self._ruby_system)
self._add_router(cluster.l2_router)

# create an ext link from a controller to a router
self.l3_router_link = RubyExtLink(ext_node=self.l3_cache, int_node=self.l3_router, bandwidth_factor=64)
self._add_ext_link(self.l3_router_link)
for cluster in self.core_clusters:
cluster.l1_router_link = RubyExtLink(ext_node=cluster.l1_cache, int_node=cluster.l1_router)
self._add_ext_link(cluster.l1_router_link)
cluster.l2_router_link = RubyExtLink(ext_node=cluster.l2_cache, int_node=cluster.l2_router)
self._add_ext_link(cluster.l2_router_link)

def _create_internal_links(self):
# create L1/L2 links
for cluster in self.core_clusters:
l1_to_l2, l2_to_l1 = RubyIntLink.create_bidirectional_links(cluster.l1_router, cluster.l2_router)
cluster.l1_to_l2_link = l1_to_l2
cluster.l2_to_l1_link = l2_to_l1
self._add_int_link(l1_to_l2)
self._add_int_link(l2_to_l1)
# create L2/main_router links
for cluster in self.core_clusters:
l2_to_main, main_to_l2 = RubyIntLink.create_bidirectional_links(cluster.l2_router, self.main_router)
cluster.l2_to_main_link = l2_to_main
cluster.main_to_l2_link = main_to_l2
self._add_int_link(l2_to_main)
self._add_int_link(main_to_l2)
# create L3/main_router link
l3_to_main, main_to_l3 = RubyIntLink.create_bidirectional_links(self.l3_router, self.main_router, bandwidth_factor=64)
self.l3_to_main_link = l3_to_main
self.main_to_l3_link = main_to_l3
self._add_int_link(l3_to_main)
self._add_int_link(main_to_l3)
Loading