Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPU] RandomUniform seed alignment #25098

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
173 commits
Select commit Hold shift + click to select a range
2a8cb94
[ADD] Multiple modes support for RandomUniform
PiotrKrzem Apr 17, 2024
2c9865a
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem May 20, 2024
05b2aeb
[ADD] All conversion methods, restructure code
PiotrKrzem May 21, 2024
12f85e4
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem May 21, 2024
f734af7
[FIX] Returned TF previous_state matches the expected state, fix key-…
PiotrKrzem May 22, 2024
4d70dad
Merge branch 'master' into feature/random_uniform_alignment
mlukasze May 23, 2024
f241832
[FIX] Clang, python
PiotrKrzem May 23, 2024
a831ef4
Merge branch 'feature/random_uniform_alignment' of https://github.com…
PiotrKrzem May 23, 2024
935000c
[FIX] Pylint pt 2
PiotrKrzem May 23, 2024
f22e922
[FIX] Flake8 linter errors
PiotrKrzem May 23, 2024
3bf5fb5
Delete .pylintdict
PiotrKrzem May 23, 2024
4a1ca0e
Merge branch 'openvinotoolkit:master' into feature/random_uniform_ali…
PiotrKrzem May 23, 2024
8bc24e3
[FIX] Address all review issues
PiotrKrzem May 23, 2024
dee2e12
[FIX] Linkage of unnamed namespaces
PiotrKrzem May 23, 2024
bcd48e0
[FIX] Remove changes to rms
PiotrKrzem May 23, 2024
720984b
[ADD] Add alignment to attributes and type_prop tests
PiotrKrzem May 23, 2024
76a2744
[ADD] Mock generator, comments and clarity for convert
PiotrKrzem May 23, 2024
eea8a90
[FIX] Clang style for enum
PiotrKrzem May 23, 2024
82f8a2c
[FIX] enum class single-line declaration
PiotrKrzem May 23, 2024
81eb774
[FIX] Variable type for emmake
PiotrKrzem May 23, 2024
164c03e
Update phillox_generator.hpp
PiotrKrzem May 23, 2024
49ec60b
Update phillox_generator.hpp
PiotrKrzem May 23, 2024
1564353
Update phillox_generator.hpp
PiotrKrzem May 23, 2024
b9df293
Update phillox_generator.cpp
PiotrKrzem May 23, 2024
b007264
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem May 29, 2024
20f56d6
[MODIFY] Remove python tests, add reference tests in C++
PiotrKrzem Jun 3, 2024
c8b0bed
[FIX] Apply recommendations from review
PiotrKrzem Jun 4, 2024
2fd50bf
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem Jun 4, 2024
7e274dd
[FIX] Enable all tests
PiotrKrzem Jun 4, 2024
fa7d5df
[FIX] Missing type specifier for constexpr
PiotrKrzem Jun 4, 2024
0cda0cd
[FIX] Override compilation error
PiotrKrzem Jun 4, 2024
f880acd
[FIX] Abs threshold for tests
PiotrKrzem Jun 10, 2024
d0785fc
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem Jun 11, 2024
daadfe8
[FIX] Standardize generated values to 4 per generation, fix pytorch o…
PiotrKrzem Jun 11, 2024
105dac3
[FIX] Ensure 64 bit representation when bit shifting
PiotrKrzem Jun 12, 2024
362e12e
[ADD] Mersenne kernel
PiotrKrzem Jun 18, 2024
2bdcaed
[ADD] Tests of exactness
PiotrKrzem Jun 18, 2024
c1839f4
[FIX] Enable x64 tests for full test coverage
PiotrKrzem Jun 18, 2024
a485dfc
[FIX] Remove OPENVINO alignment, merge OPENVINO and TENSORLFOW alignm…
PiotrKrzem Jun 27, 2024
d1e545d
[FIX]Build issues
PiotrKrzem Jun 27, 2024
a3cae3a
[FIX] flake8 error
PiotrKrzem Jun 27, 2024
128fa7f
Merge branch 'feature/random_uniform_alignment' into feature/cpu_rand…
PiotrKrzem Jun 27, 2024
894ad26
Merge branch 'master' into feature/random_uniform_alignment
mlukasze Jun 27, 2024
5122ed1
[FIX] PyTorch incorrect elem count converted
PiotrKrzem Jun 27, 2024
711b2c9
Merge branch 'feature/random_uniform_alignment' of https://github.com…
PiotrKrzem Jun 27, 2024
9a78037
[FIX] Apply suggestions from review
PiotrKrzem Jul 1, 2024
d2667d1
[FIX] Apply suggestions from review
PiotrKrzem Jul 1, 2024
da939d4
Merge branch 'feature/random_uniform_alignment' of https://github.com…
PiotrKrzem Jul 1, 2024
0c6dd8b
Merge branch 'master' into feature/random_uniform_alignment
PiotrKrzem Jul 1, 2024
1d04287
Merge branch 'feature/random_uniform_alignment' into feature/cpu_rand…
PiotrKrzem Jul 1, 2024
d695bff
[FIX] Apply namespace changes
PiotrKrzem Jul 1, 2024
f9f9427
[FIX] Move CPU check to isSupportedOp
PiotrKrzem Jul 2, 2024
12689d5
[FIX] Move CPU check to isSupportedOp
PiotrKrzem Jul 2, 2024
0adc6f1
Merge branch 'feature/random_uniform_alignment' into feature/cpu_rand…
PiotrKrzem Jul 2, 2024
2c424cd
[FIX] Const qualifier cast
PiotrKrzem Jul 2, 2024
f5ad785
Merge branch 'feature/random_uniform_alignment' into feature/cpu_rand…
PiotrKrzem Jul 2, 2024
c107516
Merge branch 'master' into feature/random_uniform_alignment
mlukasze Jul 3, 2024
fafa5ce
Merge branch 'feature/random_uniform_alignment' into feature/cpu_rand…
PiotrKrzem Jul 3, 2024
3b9844a
Merge branch 'openvinotoolkit:master' into feature/cpu_random_uniform…
PiotrKrzem Jul 3, 2024
31e0e65
[FIX] Code clarity, reorganize structure, change default behavior to …
PiotrKrzem Jul 3, 2024
709a595
[FIX] Remove OPENVINO mode from tests
PiotrKrzem Jul 3, 2024
4ca5927
[FIX] Refactor to match correct naming
PiotrKrzem Jul 8, 2024
8ae5524
[FIX] Compilation errors, temporalily disable PYTORCH tests to ensure…
PiotrKrzem Jul 8, 2024
008457f
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Jul 8, 2024
0bb7ff4
[FIX] Clang
PiotrKrzem Jul 8, 2024
19138b7
[FIX] Rename phillox -> philox of hpp files
PiotrKrzem Jul 9, 2024
5053e17
[FIX] Enable Pytorch tests
PiotrKrzem Jul 9, 2024
af054e2
[FIX] Stylistic errors
PiotrKrzem Jul 9, 2024
c7a2913
[FIX] Add missing refactor
PiotrKrzem Jul 9, 2024
b32227f
[FIX] Clang
PiotrKrzem Jul 9, 2024
8fb7e1c
[FIX] Fix GPU tests
PiotrKrzem Jul 9, 2024
1639088
[FIX] Macos linter errors
PiotrKrzem Jul 9, 2024
2b9884f
[FIX] More macos errors
PiotrKrzem Jul 9, 2024
8e80e7b
[FIX] Macos x64 hpp kernel
PiotrKrzem Jul 9, 2024
8dbce7a
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Jul 9, 2024
bd5a139
Merge branch 'master' into bugfix/refactor_random_uniform
PiotrKrzem Jul 9, 2024
7b0bac8
Merge branch 'bugfix/refactor_random_uniform' of https://github.com/P…
PiotrKrzem Jul 9, 2024
714dd90
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Jul 9, 2024
6243fa8
Merge branch 'bugfix/refactor_random_uniform' into feature/cpu_random…
PiotrKrzem Jul 11, 2024
e13f353
[FIX] Modify, improve and fix all Mersenne errors
PiotrKrzem Jul 19, 2024
470b326
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Jul 19, 2024
668447b
[FIX] Clang
PiotrKrzem Jul 19, 2024
0d00d7e
[FIX] All compilation errors
PiotrKrzem Aug 13, 2024
01880b9
[FIX] Apply recommended changes
PiotrKrzem Aug 20, 2024
81be70d
[FIX] Reduce the number of used Vmms, optimize MersenneTwister implem…
PiotrKrzem Aug 20, 2024
4739123
[FIX] Split converter function for each data type
PiotrKrzem Aug 20, 2024
23092e8
[FIX] Remove unused registers, fix compilation error
PiotrKrzem Aug 20, 2024
71f4490
[FIX] Add missing dtype conversions
PiotrKrzem Aug 20, 2024
3ff91b1
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Aug 21, 2024
629ace6
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Aug 21, 2024
1677c50
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Aug 26, 2024
4ba90d8
[FIX] CI, optimize time conversion with compile time params
PiotrKrzem Aug 27, 2024
d66f1e4
[FIX] std min type deduction
PiotrKrzem Aug 27, 2024
15208fc
[FIX] Parallel threads dtype
PiotrKrzem Aug 27, 2024
e04ad80
[FIX] Remove auto from threads_num for CI comparison
PiotrKrzem Aug 27, 2024
f6527ad
[FIX] Build error on GPU
PiotrKrzem Aug 27, 2024
25ab168
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Sep 3, 2024
3c1101f
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Sep 3, 2024
7a0f542
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Sep 23, 2024
cfdfa4c
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Oct 18, 2024
301a47a
[FIX] Spaces alignment
PiotrKrzem Oct 18, 2024
c89246b
[FIX] Apply changes from review
PiotrKrzem Oct 18, 2024
a33d79c
[FIX] Memory leaks, optimize int64 with RDX register divison
PiotrKrzem Oct 20, 2024
a53f81b
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Oct 21, 2024
d4de82d
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Oct 22, 2024
ebb4a18
[FIX] Unused variable, correct thread params assignment
PiotrKrzem Oct 22, 2024
34d086c
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Oct 22, 2024
77d5c60
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Oct 23, 2024
97830ff
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Oct 23, 2024
63e66aa
[FIX] Ensure all Tensorflow tests pass
PiotrKrzem Oct 24, 2024
ab706dd
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Oct 24, 2024
4c64fea
[FIX] macos build errors
PiotrKrzem Oct 24, 2024
354d12a
[FIX] Reduce variables count, improve readability
PiotrKrzem Oct 25, 2024
c868b95
[FIX] Split initVectors to SSE/AVX2, AVX512
PiotrKrzem Oct 26, 2024
75308df
[FIX] InitVectors segfault
PiotrKrzem Oct 26, 2024
685f9e1
[FIX] process, generateRandomNumbers, correect initialization of stat…
PiotrKrzem Oct 26, 2024
76ca56c
[FIX] Refactor AVX512 conversion of floats
PiotrKrzem Oct 26, 2024
3b5506e
[FIX] Conversion of 64 bit values
PiotrKrzem Oct 26, 2024
24d7be8
[FIX] Remove unused register
PiotrKrzem Oct 26, 2024
5983cb9
[FIX] CPU Ref any, refactor conversion, remove 64 bit support
PiotrKrzem Oct 27, 2024
a7c6907
[FIX] CPU Ref any, refactor conversion, remove 64 bit support
PiotrKrzem Oct 27, 2024
5b4bd3e
[FIX] Fix storage of uneven split of Vmm
PiotrKrzem Oct 28, 2024
fe3b9a9
[FIX] inserti128 xmm input
PiotrKrzem Oct 28, 2024
0846072
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Oct 28, 2024
1978d2c
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Oct 28, 2024
f0bd6a3
[FIX] AVX2 division error
PiotrKrzem Oct 28, 2024
3fe18d5
[FIX] Conversion for SSE41, AVX512
PiotrKrzem Oct 28, 2024
6f21e04
[FIX] int -> uint64_t type safety
PiotrKrzem Oct 28, 2024
bef3588
[FIX] Enable all tests, fix trailing whitespace
PiotrKrzem Oct 28, 2024
eecbf8a
[FIX] Remove debug error print from kernel
PiotrKrzem Oct 28, 2024
335dfd7
[FIX] Remove thread debug prints
PiotrKrzem Oct 28, 2024
84d306d
[FIX] Macos cpplint
PiotrKrzem Oct 28, 2024
24e1f35
[FIX] Formatting
PiotrKrzem Oct 28, 2024
5f3cf89
[FIX] Blank lines macos
PiotrKrzem Oct 28, 2024
e023a71
[FIX] Inference on AVX512
PiotrKrzem Oct 28, 2024
a7ee02e
[FIX] AVX512 rounding
PiotrKrzem Oct 29, 2024
7537fcf
[FIX] Cleanup of comments
PiotrKrzem Oct 29, 2024
bdb5976
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Oct 29, 2024
a6c739e
[FIX] Reduce Registers usage for SSE41, rename variables for clarity
PiotrKrzem Oct 29, 2024
3abea39
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Oct 29, 2024
87968e4
[FIX] pshufd bitmask for SSE41 mismatch of results
PiotrKrzem Oct 29, 2024
f18cf0e
[FIX] std::max macos error
PiotrKrzem Oct 29, 2024
d1bab2a
[FIX] std::max int64_t from long
PiotrKrzem Oct 29, 2024
64edc62
[FIX] Remove debug artifacts
PiotrKrzem Oct 29, 2024
bfa74e4
[FIX] Aesthetics, reverse variable names for Philox
PiotrKrzem Oct 29, 2024
e665ca1
[FIX] Remove bitshifts for stability on platforms
PiotrKrzem Oct 30, 2024
3553f96
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Oct 30, 2024
f220fde
[DEBUG] Disable parallelism on machines
PiotrKrzem Oct 30, 2024
10d0a19
[DEBUG] Prints
PiotrKrzem Oct 30, 2024
fe642fe
[FIX] Remove debug prints, add skip test for SSE41
PiotrKrzem Oct 30, 2024
c243dca
[DEBUG] Enable test prints again
PiotrKrzem Oct 30, 2024
2fdfbd3
[FIX] Revert to before debug prints
PiotrKrzem Oct 30, 2024
deaf638
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Oct 30, 2024
63be00e
[FIX] Revert skip config changes
PiotrKrzem Oct 30, 2024
a592584
[FIX] Remove more debugging artifacts
PiotrKrzem Oct 30, 2024
e409518
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Nov 7, 2024
ce32dd5
[FIX] Apply recommendations from review, fix dst->src start idx
PiotrKrzem Nov 7, 2024
f995e46
[FIX] Remove comments, fix alignment
PiotrKrzem Nov 7, 2024
aca017a
[FIX] Revert unavailable instruction for AVX2
PiotrKrzem Nov 7, 2024
4ecacb0
[FIX] Revert uni_vmovdqu, add comment
PiotrKrzem Nov 8, 2024
ecbae4c
[FIX] Arm index calculations
PiotrKrzem Nov 8, 2024
1399ce0
[FIX] ARM64 index fix v2
PiotrKrzem Nov 8, 2024
d2c3bd6
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Nov 28, 2024
127cd50
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Dec 6, 2024
43a7fb8
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Dec 13, 2024
de2f0e1
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Dec 13, 2024
5359020
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Dec 17, 2024
7386db5
[FIX] Revert key names:
PiotrKrzem Dec 17, 2024
8a99ab7
[FIX] Clang for names
PiotrKrzem Dec 17, 2024
6059e83
Merge branch 'feature/cpu_random_uniform_alignment' of https://github…
PiotrKrzem Dec 17, 2024
6daf94b
Merge branch 'master' into feature/cpu_random_uniform_alignment
PiotrKrzem Dec 18, 2024
0869e36
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Dec 19, 2024
5cb40b0
Merge branch 'master' into feature/cpu_random_uniform_alignment
mlukasze Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
895 changes: 814 additions & 81 deletions src/plugins/intel_cpu/src/nodes/kernels/x64/random_uniform.cpp

Large diffs are not rendered by default.

92 changes: 87 additions & 5 deletions src/plugins/intel_cpu/src/nodes/kernels/x64/random_uniform.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,18 @@
namespace ov {
namespace intel_cpu {
namespace kernel {
namespace random_uniform {

struct RandomUniformCompileParams {
struct PhiloxGeneratorCompileParams {
element::Type out_data_type = element::f32;
};

struct RandomUniformCallArgs {
struct MersenneTwisterGeneratorCompileParams {
element::Type out_data_type = element::f32;
bool optimized = false;
};

struct PhiloxGeneratorCallArgs {
void* dst_ptr;
const void* key_ptr;
const void* counter_ptr;
Expand All @@ -26,12 +32,23 @@ struct RandomUniformCallArgs {
uint64_t work_amount = 0lu;
};

struct MersenneTwisterGeneratorCallArgs {
PiotrKrzem marked this conversation as resolved.
Show resolved Hide resolved
void* dst_ptr;
void* state_ptr;
const void* min_ptr;
const void* range_ptr;
uint64_t output_idx = 0;
uint64_t max_output_idx = 0;
uint64_t state_accesses_count = 0lu;
int64_t elements_to_generate = 0lu;
};

template <dnnl::impl::cpu::x64::cpu_isa_t isa>
class RandomUniform : public JitKernel<RandomUniformCompileParams, RandomUniformCallArgs> {
class PhiloxGenerator : public JitKernel<PhiloxGeneratorCompileParams, PhiloxGeneratorCallArgs> {
public:
DECLARE_CPU_JIT_AUX_FUNCTIONS(RandomUniform)
DECLARE_CPU_JIT_AUX_FUNCTIONS(PhiloxGenerator)

explicit RandomUniform(const RandomUniformCompileParams& jcp);
explicit PhiloxGenerator(const PhiloxGeneratorCompileParams& jcp);

void generate() override;

Expand Down Expand Up @@ -102,6 +119,71 @@ class RandomUniform : public JitKernel<RandomUniformCompileParams, RandomUniform
static constexpr uint64_t STATISTIC_MAXIMIZING_MULTIPLIER_COUNTER = 0xCD9E8D57;
};

template <dnnl::impl::cpu::x64::cpu_isa_t isa>
class MersenneTwisterGenerator
: public JitKernel<MersenneTwisterGeneratorCompileParams, MersenneTwisterGeneratorCallArgs> {
public:
DECLARE_CPU_JIT_AUX_FUNCTIONS(MersenneTwisterGenerator)

explicit MersenneTwisterGenerator(const MersenneTwisterGeneratorCompileParams& jcp);

void generate() override;

private:
using Vmm = typename dnnl::impl::utils::conditional3<isa == dnnl::impl::cpu::x64::avx512_core,
Xbyak::Zmm,
isa == dnnl::impl::cpu::x64::sse41,
Xbyak::Xmm,
Xbyak::Ymm>::type;
using Vmask = typename dnnl::impl::utils::conditional3<isa == dnnl::impl::cpu::x64::avx512_core,
Xbyak::Opmask,
isa == dnnl::impl::cpu::x64::sse41,
Xbyak::Xmm,
Xbyak::Ymm>::type;

RegistersPool::Reg<Xbyak::Reg64> r64_dst;
RegistersPool::Reg<Xbyak::Reg64> r64_state;
RegistersPool::Reg<Xbyak::Reg64> r64_state_accesses_count;
RegistersPool::Reg<Xbyak::Reg64> r64_elements_to_generate;
RegistersPool::Reg<Xbyak::Reg64> r64_storage_capacity;
RegistersPool::Reg<Xbyak::Reg64> r64_output_idx;
RegistersPool::Reg<Xbyak::Reg64> r64_max_output_idx;

const Xbyak::Reg64 r64_params = Xbyak::Reg64(dnnl::impl::cpu::x64::abi_param_regs[0]);

// Vector registers for input storage.
RegistersPool::Reg<Vmm> v_dst;
RegistersPool::Reg<Vmm> v_state;
RegistersPool::Reg<Vmm> v_min;
RegistersPool::Reg<Vmm> v_range;

// Vector registers for generation.
RegistersPool::Reg<Vmm> v_const_1; // for MT_CONST_1
RegistersPool::Reg<Vmm> v_const_2; // for MT_CONST_2

// Vector registers for conversion.
RegistersPool::Reg<Vmm> v_mask;
RegistersPool::Reg<Vmm> v_divisor;

// Output vector register
RegistersPool::Reg<Vmm> v_result;

void initVectors();

void process();

void generateRandomNumbers();

void convertToOutputTypeMersenne();

void storeResults();

// Mersenne Twister constants
static constexpr uint32_t MT_CONST_1 = 0x9D2C5680;
static constexpr uint32_t MT_CONST_2 = 0xEFC60000;
};

} // namespace random_uniform
} // namespace kernel
} // namespace intel_cpu
} // namespace ov
Expand Down
Loading
Loading