Fix: unrecognised host compiler flags passed to nvcc #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
YixinSong-e
approved these changes
Mar 7, 2024
hodlen
added a commit
that referenced
this pull request
Mar 13, 2024
hodlen
added a commit
that referenced
this pull request
Mar 17, 2024
* name ffn tensors properly * debug: add debug printings * refactor mul_mat and axpy subgraph in sparse ffn * Revert "debug: add debug printings" This reverts commit ade011ff82a37ba29220f0f81f097d293cf9eec7. * bugfix in computational graph * support basic full gpu offloading for mul_mat and axpy * wip: sum gpu_idx indicator * calculate gpu_idx sum on the fly * wip: code refactoring * remove gpu axpy impl duplicate * axpy without gpu_bucket * minor: clean dead code * minor on comments * refactor: mul_mat and axpy should not return NULL * remove unsed lock * refactor: better naming for mul_mat_idx * refactor: separate sparse mul_mat from mul_mat_q * use mul_mat_idx at full gpu * refactor: reorg sparse mul_mat cuda host code * support full GPU comp of mul_mat_idx * refactor: remove llama_dense * refactor: add new opcode MUL_MAT_SPARSE * fix: CPU decoding for MUL_MAT_SPARSE * fix bugs on full-gpu computing * use op_params to mark sparse mul_mat/axpy fully offloaded or not * fix: disable cuda sync * minor bugfix * chore: gpu perf timing * refactor: def of gpu split structures * fix: unknown host compiler flags passed to nvcc (#161) * calc gpu_idx sum at load time * refactor sparse ffn building and bugfix * wip: more assersions * wip * fix: access invalid data ptr at MUL_MAT_IDX cpu op * fix: hidden bug when sparsity idx is computed on GPU * fix: ffn split when offload_ratio=0 * fix: splitting ffn when tensor offlloading incomplete * fix: bugs in CPU-GPU tensor interplay * fix: row_lookup pointer * minor: refactoring CUDA host code * minor refactor and bugfix on comp. graph * optimize: hybrid threading off on default; remove cuda sync * fix: offloading merged tensor * add assertion on n_threads for hybrid inference * fix: GPU-CPU sync issue * improve ffn input tensor placement for lower CPU-GPU sync overhead
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.