Unclaimed project
Are you a maintainer of ScaleLLM? Claim this project to take control of your public changelog and roadmap.
Changelog
ScaleLLM
A high-performance inference system for large language models, designed for production environments.
cudaefficiencygpuinferencellamallama3+8
Last updated 8 months ago
Back to changelogNew
v0.2.3
What's Changed
- misc: remove legacy logic to support quantization for other types. by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/350
- upgrade pytorch to 2.5.1 by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/351
- added cuda 12.6 build image by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/353
- fix cmake version issue for manylinux image by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/354
- kernel: added attention kernel for sm80 (Happy new year!) by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/355
- ci: fix package test workflow by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/357
- kernel: refactor attention kernel for readibility by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/358
- dev: config dev container with proper extensions by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/359
- kernel: added attention bench for profiling before optimization by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/360
- kernel: added logits soft cap support for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/362
- tools: added attention traits viewer by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/363
kernel: added swizzle for shared memory to avoid bank conflict by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/364kernel: added causal, alibi, sliding window mask for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/365kernel: refactor attention kernel and add more unittests by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/366kernel: added M/N OOB handling for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/367tools: update svg build to generate small file by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/368kernel: Added attention params and tile for different input types. by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/369kernel: added mqa and gqa support for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/370kernel: added var len and paged kv cache support for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/371kernel: added varlen and pagedkv unittests for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/372kernel: added attention kernel launch by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/373kernel: added build script to generate kernel instantiations for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/374kernel: change attention input shape from [head, seq, dim] to [seq, head, dim] by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/375kernel: added head_dim=96 support for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/376kernel: optimize attention kernel performance by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/377upgrade cutlass to 3.7.0 by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/379kernel: handle kv block range for attention kernel by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/382kernel: use cp_async_zfill instead of cute::clear for oob handling by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/383kernel: seperate oob iterations for better performance. by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/384refactor: remove batch_prefill interface by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/385refactor: stop build flash_infer kernel by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/386feat: integrate in-house scale attention and use it by default by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/380kernel: only zfill k once to improve perf for attention by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/387refactor: skip flash_attn build by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/388refactor: clean up kv cache set/get apis and improve slot id calculation perf by @guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/389Full Changelog: https://github.com/vectorch-ai/ScaleLLM/compare/v0.2.2...v0.2.3
C++