Upgrade to the latest versions of Composer, Streaming, and Flash Attention (#54, #61, #118, #124)

We've upgraded all our examples to use the latest versions of Composer, Streaming, and Flash Attention. This means speed improvements, new features, and deterministic, elastic, mid-epoch resumption, thanks to our Streaming library!

The repo is now pip installable from source (#76, #90)

The repo can now be easily installed for whichever example you are interested in using. For example, to install components for the llm example, navigate to the root and run pip install -e .[llm]. We will be putting the package on PyPi soon!

Support for FSDP wrapping more HuggingFace models (#83, #106)

We've added support for using FSDP to wrap more types of HuggingFace models like BLOOM and OPT.

In-Context Learning (ICL) evaluation metrics (#116)

The ICL evaluation tools from Composer 0.12.1 are now available for measuring metrics like LAMBADA, HellaSwag, PIQA, etc. for Causal LMs. See the llm/icl_eval/ folder for templates. These ICL metrics can also be measured live during training with minimal overhead. Please see our blogpost for more details.

Simple BERT finetuning example (#141)

In addition to our example of finetuning BERT on the full suite of GLUE tasks, we've added an example of finetuning on a single sequence classification dataset. This should be a simple entrypoint to finetuning BERT compared with all the bells and whistles of our GLUE example.

NeMo Megatron example (#84, #138)

We've added a simple example of how to get started running NeMo Megatron on MCloud!