Shiperator is back

A long time ago in a galaxy not that far away... Shipyard had a Shiperator trait. This trait was a copy of std's Iterator. The problem it solved was filter iteration over storage tracking modification. filter iterates through all elements before applying the filter. This is important because back then components were flagged for modification as soon as the reference was created, before the filter ran. That meant all components would get flagged. To prevent that, Shiperator returned a shipyard::Filter. It was aware of tracking and didn't flag filtered out components. Shiperator was then removed when Mut was introduced. Instead of flagging component immediately, they only get flagged when Mut is dereferenced.

iter module rework

One of the main goal of this rework was to enable custom views and built-in views iteration.

Example:

use shipyard::{Borrow, Component, IntoIter, View, World};

#[derive(Component)]
struct FlightSpeed;
#[derive(Component)]
struct HasHorn;
#[derive(Component)]
struct IsFrozen;

#[derive(Borrow, IntoIter)]
struct PegasusView<'v> {
    v_flight_speed: View<'v, FlightSpeed>,
    v_has_horn: View<'v, HasHorn>
    // Could have multiple other fields
}

fn main() {
    let mut world = World::new();

    world.add_entity((FlightSpeed, HasHorn, IsFrozen));

    world.run(|v_pegasus: PegasusView, v_is_frozen: View<IsFrozen>| {
        // Here we want to iterate over entities that have all FlightSpeed, HasHorn and IsFrozen components
        for (pegasus, is_frozen) in (&v_pegasus, &v_is_frozen).iter() {
            let _: (Pegasus, &IsFrozen) = (pegasus, is_frozen);
        }
    });
}

(&v_pegasus, &v_is_frozen).iter() this iterator didn't work in v0.7. It was possible to iterate multiple built-in views OR a custom view but not both.

Some traits that I used to know

To create an iterator, previous versions of Shipyard relied on a few traits.

• IntoAbstract transformed references to views into structs of raw pointers. (e.g. &View<T> -> FullRawWindow<T>)
• AbstractMut used the raw pointers to generate the component references the iterator returned. (e.g. FullRawWindow<T> -> &T)
• IntoIter created the iterator struct containing the structs of raw pointers alongside the mechanism to advance the iteration. This is a replacement for std's IntoIterator. Shipyard implements this trait on tuples, since it neither owns the tuple type nor IntoIterator, it cannot be used directly.

These traits were too linked to SparseSet (Shipyard's default component storage) to be able to pull off custom views <-> built-in views iteration. There is also a future big feature (groups) that wouldn't be possible. So I decided to start over.

A new dream team

This new version also works by leveraging multiple traits.

• IntoShiperator transforms references to views into structs of raw pointers. Very similar job as IntoAbstract but simpler. Where IntoAbstract had 10 functions, IntoShiperator has 3.
• ShiperatorOutput defines the type returned by the iterator, it's a very simple trait.
• ShiperatorCaptain and ShiperatorSailor both generate the actual component references.
• Shiperator is a struct containing the structs of raw pointers alongside the mechanism to advance the iteration. There is now a single struct for all iterators generated by Shipyard. Custom views don't need to re-invent the wheel.

While it may seem more complex, each trait is more focused. The overall number of functions is down to 11 from 17 while enabling more functionalities.

Performance

Iterators from previous versions were already optimized so the goal was not to go faster but try to keep the same performance. This is why IntoAbstract was tightly coupled with SparseSet. It had functions returning internal parts to enable fast iteration.

The performance of most iterators is identical with the exception of with_id. Its implementation has completely changed which resulted in better performance.

| 10k entities iteration | 0.7 | 0.8 | Diff | | :--------------------: | :------: | :------: | :---: | | single component | 20.64 µs | 4.04 µs | -80% | | two components | 42.32 µs | 11.13 µs | -73% |

Finding a partner (when you're a storage)

Groups will be a way to get even better iteration performance at the cost of insertion/removal. If you are familiar with ECS you may recognize this trade-off as SparseSet vs archetypes and you would be correct. I will go into more details when groups are implemented.

This version introduces one of the required pieces. Given multiples views iterated together, how can grouped storages find their partners?

Most solutions involve collecting TypeIds of the iterated storages. This is not as easy as it seems. Previous versions used to have a function IntoAbstract::type_id. It works fine for single storages like View<T>. But for something like PegasusView or (&View<T>, &View<U>) it becomes impossible to return a single TypeId. A Vec<TypeId> would work but allocating in every iterator creation isn't ideal.

The current solution: dating ad. Instead of collecting all TypeIds, only storages that are in a group advertise themselves. It becomes part of the cost of groups and should be offset by the faster iteration while other storages are not impacted.

Longer tracking

Shipyard's tracking is based on a counter. It doesn't track time but the number of systems ran since the start of the World. Previous versions used a u32 counter. This allows for dozens to hundreds of hours between when a component is flagged to when it's checked.

use shipyard::{Component, View, World};

#[derive(Component)]
#[track(Insertion)]
struct A;

fn main() {
    let mut world = World::new();

    let eid = world.add_entity(A);

    // Imagine billions of systems running here

    world.run(|v_a: View<A>| {
        if v_a.is_inserted(eid) {

        }
    });
}

I used to think this would be enough. For longer tracking, users would implement another solution. But this other solution would be tricky to implement so instead the counter is now a u64.

Thanks @80ROkWOC4j for this change!

lib.rs refactor

Over the years more and more exports accumulated in lib.rs. I like to have most exports there, it makes it easy to write import shipyard::*; for small projects. It also makes it easy to find items on docs.rs, but too many exports can have the opposite effect.

With this release I tried to move as many items as possible without impacting "most use cases". The result is about half items moved to now public modules.

Extended tuple

Shipyard has many traits implemented on tuples. Since Rust doesn't have variadic tuples that means more code for each tuple and longer compile times. Historically Shipyard has had a 10 tuple size maximum. Some traits could avoid this limit by nesting but it can be cumbersome. I used to think the maintenance cost wasn't worth it but I changed my mind.

The new extended_tuple feature allows users to choose what best fits their needs.

| | no feature | extended_tuple | | :------------: | :--------: | :------------: | | Max tuple size | 10 | 32 | | Expanded LoC | 45k | 215k | | Compile time | 4s | 18s |

Also a big thanks to @notinmybackyaard for contributing multiple patches to v0.7! They also contributed the new MemoryUsageDetail trait for clearer memory usage debugging.