Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternative fallback fields implementation #441

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

LoopyAshy
Copy link

@LoopyAshy LoopyAshy commented Oct 21, 2022

This is an alternative implementation to #440.
This implementation is notably different in that it is more controllable by the implementor and feels more like a first class feature.
The main changes are the addition of types deriving Any to state the priority of dynamic fields vs actual fields. The options are as follows:

  • First - Will try to find a dynamic field first before checking static fields.
  • Last - Will try to find a static field first before checking dynamic fields.
  • Never - Will completely disregard any dynamic fields and exclusively use static fields. (Same behavior as prior to this implementation and the default)
  • Only - Will completely disregard static fields and exclusively use dynamic fields.

All of these can be applied to any type using the Derive(Any) macro by applying the attribute
#[rune(dynamic_fields="never|only|first|last")] to the struct itself.

Dynamic fields functionality is basically the same as the #440 however due to the fact First and Only exist means there needed to be a change to their implementation.
DYNAMIC_FIELD_GET now handles Options slightly differently: if you return None it will presume there was no field, Some(None) is still a option if desired. The value will be unwrapped from the Option on the callers side so doesn't need to be worried about by the script writer.
DYNAMIC_FIELD_SET is basically the same as DYNAMIC_FIELD_GET however instead of a Option you can choose to return false which will be treat as a failure to set the field, allowing it to try and set the static field next if the type is First or simply return a error otherwise as expected with static fields.
For both of the above changes to DYNAMIC_FIELD_SET and DYNAMIC_FIELD_GET: you can return any other type (including () of course) and it will not treat it any differently to before.

Examples

A struct with dynamic fields which are prioritised over static fields:

#[derive(Any, Clone)]
#[rune(dynamic_fields="first")]
struct TestStruct {
    values: HashMap<String, u32>,
    b: u32, 
    c: u32
}

implementing fallible DYNAMIC_FIELD_GET and DYNAMIC_FIELD_SET

let mut module = Module::new();

module.ty::<TestStruct>()?;
module.inst_fn(
    Protocol::DYNAMIC_FIELD_GET,
    |this: &TestStruct, key: String| {
        if let Some(v) = this.values.get(&key) {
            Some(*v)
        } else {
	    None
	}
    },
)?;
module.inst_fn(
    Protocol::DYNAMIC_FIELD_SET,
    |this: &mut TestStruct, key: String, value: u32| {
        if key == "nerd" {
            false
        } else {
            this.values.insert(key, value);
            true
	}
    },
)?;

Reasoning

Sometimes it is desired for user readability to pretend a specific custom type has fields and instead accessing the data from elsewhere such as from a HashMap or another collection.
possible use cases is having types with dynamic layouts while having rules to their usage such as locking the layout once created.
it also allows custom types to be dynamically extended by the scripter while keeping the benefits of being a static Shared<AnyObj> but not seeming like a collection which [] accesses give the impression of.

Benchmarks

Note
Due to the tiny footprints of the functions, the differences are very hard to see if any and can be improved within my benchmarks with a faster hashmap regardless but honestly values these small are easily within margin of error.

Before

test fields::actual_field_get                    ... bench:         418 ns/iter (+/- 88)
test fields::actual_field_set                    ... bench:         434 ns/iter (+/- 161)

test fields::string_key_classic_index_get        ... bench:         523 ns/iter (+/- 52)
test fields::static_string_key_classic_index_get ... bench:         486 ns/iter (+/- 80)

After

test meta_fields::actual_field_get_first             ... bench:         614 ns/iter (+/- 18)
test meta_fields::actual_field_get_last              ... bench:         420 ns/iter (+/- 46)
test meta_fields::actual_field_get_never             ... bench:         414 ns/iter (+/- 27)
test meta_fields::actual_field_set_first             ... bench:         514 ns/iter (+/- 63)
test meta_fields::actual_field_set_last              ... bench:         427 ns/iter (+/- 14)
test meta_fields::actual_field_set_never             ... bench:         408 ns/iter (+/- 28)
test meta_fields::actual_field_set_only              ... bench:         515 ns/iter (+/- 16)
test meta_fields::static_string_index_get_first      ... bench:         467 ns/iter (+/- 14)
test meta_fields::static_string_index_get_last       ... bench:         471 ns/iter (+/- 35)
test meta_fields::static_string_index_get_never      ... bench:         465 ns/iter (+/- 43)
test meta_fields::static_string_index_get_only       ... bench:         459 ns/iter (+/- 17)
test meta_fields::static_string_meta_field_get_first ... bench:         531 ns/iter (+/- 64)
test meta_fields::static_string_meta_field_get_last  ... bench:         546 ns/iter (+/- 31)
test meta_fields::static_string_meta_field_get_only  ... bench:         518 ns/iter (+/- 35)
test meta_fields::static_string_meta_field_set_first ... bench:         525 ns/iter (+/- 231)
test meta_fields::static_string_meta_field_set_last  ... bench:         545 ns/iter (+/- 42)
test meta_fields::static_string_meta_field_set_only  ... bench:         515 ns/iter (+/- 34)
test meta_fields::string_index_get_first             ... bench:         504 ns/iter (+/- 24)
test meta_fields::string_index_get_last              ... bench:         505 ns/iter (+/- 42)
test meta_fields::string_index_get_never             ... bench:         504 ns/iter (+/- 17)
test meta_fields::string_index_get_only              ... bench:         501 ns/iter (+/- 20)
test meta_fields::string_meta_field_get_first        ... bench:         552 ns/iter (+/- 21)
test meta_fields::string_meta_field_get_last         ... bench:         595 ns/iter (+/- 26)
test meta_fields::string_meta_field_get_only         ... bench:         558 ns/iter (+/- 26)
test meta_fields::string_meta_field_set_first        ... bench:         508 ns/iter (+/- 23)
test meta_fields::string_meta_field_set_last         ... bench:         553 ns/iter (+/- 39)
test meta_fields::string_meta_field_set_only         ... bench:         510 ns/iter (+/- 23)

Closes

#381 #263

Renamed to `META_FIELD_GET` and `META_FIELD_SET`.
Added new field to `AnyObjVTable` to identify if it should use meta fields.
Added new trait `MetaFieldMode` which is now required on `Any`
Made it so the `MetaFieldMode` determines how the fields are accessed on that specific type, by default it is `Never` meaning it will use the old behavior.
`Always` means it never uses `GET` or `SET`
`First` means it will check for meta fields before checking via the old behavior.
`Last` means it will check via the old behavior first before checking meta fields.
Added meta field search rules to derive `Any` macro via attribute `#[rune(meta_fields="never|first|last|only")]
Allowed META_FIELD_GET to return Option which is unwrapped before returning, if it receives None then it treats it as though the field was missing.

META_FIELD_SET instead asks for false to be returned if it was a failure to set the field.
@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

I'm gonna have to think this over. It's not a style choice that I would choose, so if I include it I'd like to see the following changes done:

  • Introduce your feature as extra Vm instructions which are feature gated. So instead of rewriting the ObjectSlotIndexGet instruction, add a new one which uses the new field loading convention.
  • Feature gate the extra plumbing in AnyObj which allows for dynamic index get conventions.
  • Add a compiler option which changes how index gets are performed so that they make use of the new feature. This would allow you to use the new feature, but it wouldn't be enabled by default.

That way the feature can be tested and refined in parallel.

Concerns

  1. Dynamically deciding on the field loading convention in use is a form of metaprogramming which doesn't obviously mesh well with a future gradual type system. 2. Having dynamic field loading conventions might make future compile time optimizations harder or not possible.

Both of these are future potentials and not current blockers, which is why I wouldn't mind including the feature on an experimental basis to get a feel for it. Maybe improve it and mainline include in the future.

Thank you!

@udoprog udoprog added enhancement New feature or request lang Issues related to language design. labels Oct 23, 2022
@LoopyAshy
Copy link
Author

I'm gonna have to think this over. It's not a style choice that I would choose, so if I include it I'd like to see the following changes done:

  • Introduce your feature as extra Vm instructions which are feature gated. So instead of rewriting the ObjectSlotIndexGet instruction, add a new one which uses the new field loading convention.
  • Feature gate the extra plumbing in AnyObj which allows for dynamic index get conventions.
  • Add a compiler option which changes how index gets are performed so that they make use of the new feature. This would allow you to use the new feature, but it wouldn't be enabled by default.

That way the feature can be tested and refined in parallel.

Concerns

  1. Dynamically deciding on the field loading convention in use is a form of metaprogramming which doesn't obviously mesh well with a future gradual type system. 2) Having dynamic field loading conventions might make future compile time optimizations harder or not possible.

Both of these are future potentials and not current blockers, which is why I wouldn't mind including the feature on an experimental basis to get a feel for it. Maybe improve it and mainline include in the future.

Thank you!

Thank you for your reply. I am happy to make those changes and will do so when I am free.

Regarding you concerns: I do believe it shouldn't make it completely impossible but I can see the possibility of increasing the complexity of the compilation, if it does come to that I am however more than happy to try and assist in the ease of implementation as this is a QoL feature that I know will be very useful in quite a few of my projects for example in one of my projects the types are generated at load time and stored as templates and generated dynamically at runtime when requested and are optimised around these facts.
Reason for this is so the project doesn't need recompiled as it relies on third party data types which are very often change but the layouts are available, however there layouts are statically typed so if I gave a float instead of a int it would be incorrect, so I have to check the typing and field name validity etc.

When I was using Lua for this project I simply overwrote the meta function for getting and setting fields and it worked like any other type and looked no different, it was seamless and effective.
When moving to rune due to it having many many preferred design decisions over lua, it was only possible by having dedicated instance functions or using INDEX_* which means using ["field_name"] which makes it look strange to the script writers who think of these types are true types not HashMaps or collections.

Renamed meta fields to dynamic fields and everything related to them.
Applied `dynamic_fields` feature gate to `rune` and `rune-macros`
Added compiler option to enable dynamic fields.
Updated tests to match.
@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 23, 2022

I have added the requested changes along with making the naming more consistent (renaming the feature to dynamic fields from meta fields)
Let me know what you think and if anything requires changes or clean up to meet the desired structure.
I have run the previous benchmarks and no changes have been found with the previous implementation.

I want to note it was actually fun to apply these changes as it allowed me to further understand the source and its flow.

Apparently the order of the imports anger `fmt`.
Copy link
Collaborator

@udoprog udoprog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very neat, and thanks for the quick turnaround!

So most of the feature flagging can be avoided. All that really needs to be feature flagged is:

  • The option that enables the use of the different field get/set instructions in Options (see comment).
  • The enum variants in DynamicFieldMode which are not Never (see comment).
  • Use of those other variants when deriving Any in rune-macros (and the appropriate error if the feature is not enabled).
  • The AnyObj runtime support (just nice to avoid the vtable increase if it's not used but not super important). see comment.

It is perfectly all right for the instructions to exist alongside other instructions in the Vm. As long as their use is behind an option.

@@ -18,7 +22,32 @@ pub use rune_macros::Any;
/// name: String,
/// }
/// ```
pub trait Any: Named + MetaFieldMode + std::any::Any {
#[cfg(not(feature = "dynamic_fields"))]
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, avoid feature gating two incompatible traits of Any. What can happen is that you have coherence issues under a scenario such as:

Crate a defines a type that implements Any without activating the dynamic_fields feature, so they leave FIELD_MODE undefined.

Crate b defines a type that implements Any while activating the dynamic_fields feature, so they define FIELD_MODE.

Crate c depends on a and b, so the dynamic_fields feature is activated, causing a to fail to build.

Instead:

  • Unconditionally include FIELD_MODE (and possibly rename to DYNAMIC_FIELD_MODE :)).
  • Feature gate the other variants of DynamicFieldMode which are expected to cause alternate behaviors (First, Last, and Only).
  • Mark DynamicFieldMode as #[non_exhaustive]. This prevents dependents which have not enabled the feature from using it in a manner which is incompatible with the feature being enabled.

@@ -61,6 +65,10 @@ impl Options {
Some("v2") => {
self.v2 = it.next() != Some("false");
}
#[cfg(feature = "dynamic_fields")]
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be confusing if it fails to parse despite being in the documentation.

Instead:

  • Return a special parse error indicating that the option exists, but it's not supported without enabling a feature; or,
  • Emit a warning through Diagnostics during a build if the dynamic_fields option is used but the feature is disabled.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call.

#[cfg(feature = "dynamic_fields")]
let meta_fields = Some(quote! {Never});
#[cfg(not(feature = "dynamic_fields"))]
let meta_fields = None;
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some additional simplifications in rune-macros would be possible when we go for the other feature gating strategy I'm proposing. So I'm not gonna comment on them now ;).

@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

The AnyObj runtime support (just nice to avoid the vtable increase if it's not used but not super important).

Actually, no this can't be done. AnyObjVtable is public API! So it would suffer the same issues as the Any trait. The field needs to be unconditionally included.

Applied the requested changes to be more sane and heavily simplify the feature gating for `dynamic_fields`.
Reduced odds of making a mistake while reporting deactivated dynamic field features.
Also added feature flags to tests.
@LoopyAshy
Copy link
Author

I have applied the requested changes, did take me a while due to how long I been coding today lol so sorry if I made any silly mistakes but all the tests passed and the benches are stable.

@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

Don't sweat it! And don't feel like you have to make requested changes immediately. I don't have any plans to change things in Rune for at least a week or two ;).

@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 23, 2022

Don't sweat it! And don't feel like you have to make requested changes immediately. I don't have any plans to change things in Rune for at least a week or two ;).

Oh it is not for that reason lol, I just like getting stuff sorted, and when I get stuck in I usually stay focused on the task for good while. I am just glad it is all going smoothly; this is the first public repo I ever made a PR for due to confidence issues honestly, private is another thing all together though :D.

Copy link
Collaborator

@udoprog udoprog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good, a few nits!

@@ -50,7 +50,7 @@
//! [rune]: https://github.com/rune-rs/rune

use anyhow::{anyhow, Result};
use rune::compile::ParseOptionError;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to rename the error type, you've only turned it into an enum which is still one error (one of its variants). "Errors" gives the impression that it's a collection.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem I can change that easily.

self.errors.push(syn::Error::new_spanned(
s,
"attempted to set dynamic_fields to `only` while the feature `dynamic_fields` is disabled."
));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation here seems a bit off?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Funnily enough it looked perfect until cargo fmt decided to make it look like that lol, I'll appease it shortly.

attrs.meta_fields = Some(match s.value().as_str() {
"never" => format_ident!("Never"),
"only" => {
if cfg!(feature = "dynamic_fields") {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this kind of test is repeated three times it might be worth breaking out into a function.


/// Allow use of META_FIELD_GET and META_FIELD_SET
#[cfg(feature = "dynamic_fields")]
pub dynamic_fields: bool,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pub(crate) since it has a mutator method (else we can set it directly).

The mutator method can be feature gated. This field does not need to be.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking that but I wasn't 100%.

required_feature: "dynamic_fields",
});
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if cfg!(feature = "dynamic_fields") { /* */ } else { /* */ } is probably cleaner here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if cfg!(feature = "dynamic_fields") { /* */ } else { /* */ } is probably cleaner here.

Yeah easily sorted; it is just like this due to ObjectIndexDynamicGet originally being feature gated so it wouldn't compile with that if statement lol (kind of wish rust had a macro which mimicked if else cfg macro but actually ignored the invalid branch since its more sane to read.

},
span,
);
#[cfg(not(feature = "dynamic_fields"))]
c.asm.push(Inst::ObjectIndexSet { slot }, span);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving into a function which is feature gated with two different implementations instead?

Block-level feature gates are a bit awkward to deal with in my experience.

Something like:

push_field_lookup(c, slot);

},
span,
);
#[cfg(not(feature = "dynamic_fields"))]
c.asm.push(Inst::ObjectIndexGet { slot }, span);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

/// Search for a field by `String`
pub const DYNAMIC_FIELD_GET: Protocol = Protocol {
name: "dynamic_field_get",
hash: Hash::new(0x6dda58b140dfeaf9),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how did you generate these?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how did you generate these?

I used the Inst hasher and used a name which wouldn't be allowed as a field name.

CallResult::Unsupported(target) => CallResult::Unsupported(target),
}
}
#[cfg(feature = "dynamic_fields")]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a huge fan of these big featured off blocks but not much to be done about them either. Maybe consider if these sections can be cleaned up a bit (somehow) in the future. But I'm not sure how right now.

}
} else {
CallResult::Ok(result)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preferred style in instances like thse would be this to save some of the indentation:

let result = match result {
    Value::Option(result) => result,
    _ => return Ok(CallResult::Ok(result)),
};

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally I agree, was just bit tired during the coding session lol.

@LoopyAshy
Copy link
Author

I'll add the requested changes when I get back later as I am busy today, thank you for the review <3

Renamed `ParseOptionsErrors` back to `ParseOptionError`.
Cleaned feature gating in `options.rs`.
Added new feature instead of it being default (mostly as it was messing up compilation checks).
Improved indentation and cleaned up readability.
Moved the feature gated logic within `vm.rs` to macros as a readability test.
Moving the logic to a function caused borrow errors.

Not sure if this will be kept but its worth testing.
@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 28, 2022

I been busy past few days but I added the requested changes and forgot to mention it lol, I also added a tiny idea for the clean up but admittedly I ain't the most fond of it and I am happy to revert that change if you feel it is too out of place, honestly though it is quite fiddly to clean up admittedly.

I have been using these changes on one of my projects to test the usability and ease of implementation etc, and honestly it has been a very smooth experience and I haven't ran into any problems yet, which is a wonderful sentence to type lol.

Fixed dynamic field tests.
@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 28, 2022

My bad, I genuinely completely forgot to push my changes to tests. I am surprised I didn't notice sooner as I had them ready but I ran the tests myself and completely blanked out lol.

Fixed now <3

Honestly though I should really clean the tests up later as they are horrible to read and refactor.

@udoprog
Copy link
Collaborator

udoprog commented Oct 30, 2022

I'll be back after this weekend and then I'll take a look!

Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request lang Issues related to language design.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants