Improving performance of the SetTransmit hook.

[vinci6k]

Would it be possible to create this hook on the C++ side of SP, similar to the way OnPlayerRunCommand was made?

I feel like SetTransmit is one of the most expensive functions to hook, but it can be such a useful tool. Maybe if the filtering was done on the C++ side, the performance would improve?

[jordanbriere]

Please elaborate on how you would see its usage?

If you need hide only certain entity for everyone, you can use just
entity.effects |= EntityEffects.NODRAW

You can also probably avoid make_object (I don't know how expensive it is) using raw pointers to get values:
if you need only player index: index_from_pointer()
Generally: (pointer + offset).get_*()
Yes, it is very sad that we should think about perfomance and not do the real thing.

[vinci6k]

Please elaborate on how you would see its usage?

If we were to stay true to the function as much as possible, something like this:

# Specify which entity to look for within the decorator.
@OnSetTransmit('prop_physics_override')
def on_set_transmit(entity, info):
    """Called when an entity is being transmitted to a client.

    Args:
        entity (Entity): Instance of Entity that's being transmitted.
        info (CheckTransmitInfo): Instance of CheckTransmitInfo.

    """

Maybe it can be optimized further if we ditch creating the Entity instance and just pass the index instead (since most plugins will use some sort of list/set/dict to check for entities). And seeing as CCheckTransmitInfo returns an edict when accessing the client attribute, perhaps it should be converted to the player's index or userid?

Edit: Now that I think about it, creating the Entity instance would be required (or at least BaseEntity) in order to filter out the entities by classname..

[jordanbriere]

Yes, it is very sad that we should think about perfomance and not do the real thing.

If the tools that are made freely available to you make you sad, then perhaps you should stop torturing yourself and write your owns. Everyone deserves to be happy!

Please elaborate on how you would see its usage?

If we were to stay true to the function as much as possible, something like this:

# Specify which entity to look for within the decorator.
@OnSetTransmit('prop_physics_override')
def on_set_transmit(entity, info):
    """Called when an entity is being transmitted to a client.

    Args:
        entity (Entity): Instance of Entity that's being transmitted.
        info (CheckTransmitInfo): Instance of CheckTransmitInfo.

    """

Maybe it can be optimized further if we ditch creating the Entity instance and just pass the index instead (since most plugins will use some sort of list/set/dict to check for entities). And seeing as CCheckTransmitInfo returns an edict when accessing the client attribute, perhaps it should be converted to the player's index or userid?

Edit: Now that I think about it, creating the Entity instance would be required (or at least BaseEntity) in order to filter out the entities by classname..

Hmm. The optimization would be minimal and would really just be beneficial when there are multiple callbacks registered (as the wrapped pointers would be re-used for all of them). An optimal implementation would be to let the c++ callback do the filtering rather than having to interrogate the Python interpreter every calls.

An idea could be to add the following methods:

BaseEntity.hide() # Globally hide this entity for all players.
BaseEntity.unhide() # Globally unhide this entity for all players.
BaseEntity.is_hidden() # Return whether this entity is globally hidden.
Player.hide_entity(BaseEntity) # Hide the given entity for this player.
Player.unhide_entity(BaseEntity) # Unhide the given entity for this player.
Player.is_entity_hidden(BaseEntity) # Return whether the given entity is hidden for this player.

Then have an internal system that filter the SetTransmit calls based on a global map of bitsets, etc. That way the c++ callback would only have to do something like this:

def set_transmit(entity, player):
    if entity is globally hidden:
        return False

    if entity is hidden for player:
        return False

Basically, you flag an entity as hidden once and the c++ callback take care of it from there, etc. Of course this is just theoretical design but this would be optimal in my opinion because the hook already know what's up removing the back and forth, etc. At least, definitely sounds like a more worthy optimization overall.

[Ayuto]

I like your suggestion!

This would work unless you want to conditionally hide or unhide entities. So, additionally to your suggestion we might want to still call Python callbacks if there are any.

[jordanbriere]

I like your suggestion!

This would work unless you want to conditionally hide or unhide entities. So, additionally to your suggestion we might want to still call Python callbacks if there are any.

Yes, if there are callbacks registered then it makes sense to call them from there and re-use the objects. I would however vote against doing any classname filtering and directly pass entity, player to them. For example:

def set_transmit(entity, player):
    if entity is globally hidden:
        return False

    if entity is hidden for player:
        return False

    for callback in callbacks:
        if not callback(entity, player):
            return False

I don't think any other info is relevant, is it?

[vinci6k]

Basically, you flag an entity as hidden once and the c++ callback take care of it from there, etc. Of course this is just theoretical design but this would be optimal in my opinion because the hook already know what's up removing the back and forth, etc. At least, definitely sounds like a more worthy optimization overall.

This is so much better than my initial idea!
Another thing that we need to keep in mind is the FL_EDICT_ALWAYS flag. Entities that have it can't be hidden until it is stripped.

[jordanbriere]

This is so much better than my initial idea!
Another thing that we need to keep in mind is the FL_EDICT_ALWAYS flag. Entities that have it can't be hidden until it is stripped.

Perhaps you could write a draft in Python that handle such scenarios and edge cases that would have to be kept in mind? This sure would help immensely to get a better view of the big picture if all the special cases are known ahead of time as they could certainly influence how the whole system would need to be implemented. :)

[vinci6k]

Sure thing! Sorry for taking so long to respond, I've been away from home for a while. I'll post the draft in an hour or so.

[vinci6k]

# ../set_transmit/set_transmit.py

# Source.Python
from entities import CheckTransmitInfo
from entities.entity import Entity
from entities.helpers import index_from_pointer, edict_from_index
from entities.hooks import EntityPreHook, EntityCondition
from players import PlayerGenerator
from players.entity import Player
from players.helpers import userid_from_edict


FL_EDICT_ALWAYS = 1<<3

is_prop_physics = EntityCondition.equals_entity_classname(
    'prop_physics_multiplayer')

hidden_entities = {}


class EntityST(Entity):
    caching = True

    def hide(self):
        # Is this entity supposed to skip the SetTransmit hook?
        if self.edict.state_flags & FL_EDICT_ALWAYS:
            # Strip the FL_EDICT_ALWAYS flag to make the entity go through the
            # hook at least once.
            self.edict.state_flags = self.edict.state_flags ^ FL_EDICT_ALWAYS
        
        hidden_entities[self.index] = set()

    def unhide(self):
        try:
            hidden_entities.pop(self.index)
        except KeyError:
            return

    @property
    def is_hidden(self):
        # Wasn't sure how to handle this. Should it return True only if the
        # entity is hidden globally? Or even if it's hidden from one or more
        # players?
        return True if self.index in hidden_entities else False


class PlayerST(Player):
    caching = True

    def hide_entity(self, index):
        try:
            userids = hidden_entities[index]
            # Is the entity hidden globally?
            if not userids:
                return

        except KeyError:
            hidden_entities[index] = set()

        edict = edict_from_index(index)
        # Is this entity supposed to skip the SetTransmit hook?
        if edict.state_flags & FL_EDICT_ALWAYS:
            # Strip the FL_EDICT_ALWAYS flag to make the entity go through the
            # hook at least once.
            edict.state_flags = edict.state_flags ^ FL_EDICT_ALWAYS

        hidden_entities[index].add(self.userid)

    def unhide_entity(self, index):
        try:
            userids = hidden_entities[index]
        except KeyError:
            return

        # Is the entity hidden globally?
        if not userids:
            for edict in PlayerGenerator():
                userids.add(userid_from_edict(edict))

        userids.remove(self.userid)

    def is_entity_hidden(self, index):
        try:
            userids = hidden_entities[index]
        except KeyError:
            return False

        # Is the entity hidden globally or hidden from the player?
        if not userids or self.userid in userids:
            return True

        return False


@EntityPreHook(is_prop_physics, 'set_transmit')
def set_transmit_pre(stack_data):
    index = index_from_pointer(stack_data[0])

    try:
        userids = hidden_entities[index]
    except KeyError:
        return

    edict = edict_from_index(index)
    # Check if the FL_EDICT_ALWAYS flag has been reapplied.
    if edict.state_flags & FL_EDICT_ALWAYS:
        # If so, strip it again.
        edict.state_flags = edict.state_flags ^ FL_EDICT_ALWAYS

    # No userids found - hide the entity globally.
    if not userids:
        return False

    info = CheckTransmitInfo._obj(stack_data[1])
    # Should the entity be hidden from this player?
    if userid_from_edict(info.client) in userids:
        return False

From my experience with the SetTransmit hook, FL_EDICT_ALWAYS is the only special case I've come across so far. Maybe entities with the FL_EDICT_DONTSEND flag could be another? I'm not sure though, I never really messed with that flag.

[jordanbriere]

Alright so, I had some time this afternoon to play around with the idea and I ended up with something. Keep in mind this is a first pass so it is likely to change if/when I revisit the concept: 14a38d4

First of all, I discarded the idea of using SetTransmit altogether for many reasons. The first one being that it is virtual, and constantly having to check if we have a hook registered for every entities was getting redundant. Another reason is that it was very unreliable. Trying to interfere with the decision process of the engine was a pain and would have required hooks on UpdateTransmitState, ShouldTransmit, and probably some others as well as SetTransmit in order have an uniform implementation that covers all cases. So yeah, instead I opted for a post transmission filtering system that waits for the engine to be done processing the entities and then we start filtering.

For the sake of example, let's say we wanted to hide all prop physics from all players.

The main class we are going to use is TransmitFilter:

TransmitFilter(type=TransmitType.OUT, criteria=None, override=None)

The type parameter defines the type of filtering we want to do. Either IN where we want to force the transmission of an entity, or OUT where we want to block the transmission.

To hide props from all players, we would need to register an OUT filter like so:

from entities.transmit import TransmitFilter
from entities.transmit import TransmitType

@TransmitFilter(TransmitType.OUT)
def transmit_filter(entity, player):
    return entity.classname == 'prop_physics_multiplayer'

The question we have to ask ourselves when returning from a filter is: Do we want to filter this entity <type> of the transmission? In our example, the answer is; yes, we want to filter the prop out of transmission.

However, this is very inefficient to have to constantly compare classnames, etc. and this is where the criteria comes into play. It allows us to narrow down the calls to our filter to precisely what we are looking for. For example, we would use a criteria that matches prop's classname:

from entities.transmit import TransmitCriteria
from entities.transmit import TransmitFilter

is_prop_physics = TransmitCriteria.equals_entity_classname(
    'prop_physics_multiplayer')

@TransmitFilter(criteria=is_prop_physics)
def transmit_filter(entity, player):
    return True

The criteria are basically a set of states that is tested when the entities are created so there is no dynamic check of the class name as it was already done and stored. Now our filter is only called for prop physics so we can just return True. However, calling the callback just for it to return True in all cases is inefficient and we should rather set the override like so:

from entities.transmit import TransmitCriteria
from entities.transmit import TransmitFilter

is_prop_physics = TransmitCriteria.equals_entity_classname(
    'prop_physics_multiplayer')

@TransmitFilter(criteria=is_prop_physics, override=True)
def transmit_filter(entity, player):
    pass

Now our callback will never be called and all props will be filtered out of the transmission list. Having a pass function is a bit weird, so the following syntax would most likely be preferred:

TransmitFilter(criteria=is_prop_physics, override=True).initialize()

Now we don't have a callback but a default value that is used when the entity match the criteria. Now let's say we wanted to hide/unhide entities whenever we want we could do:

from entities.transmit import BaseTransmitCriteria
from entities.transmit import TransmitFilter

hiddens_entities = BaseTransmitCriteria()
TransmitFilter(criteria=hiddens_entities, override=True).initialize()

And now we could just set any entities to True whenever we feel like it. For example:

from events import Event
from players.entity import Player

@Event('player_say')
def player_say(game_event):
    player = Player.from_userid(game_event.get_int('userid'))
    entity = player.view_entity
    if game_event['text'] == 'hide':
        hiddens_entities[entity.index] = True
    else:
        hiddens_entities[entity.index] = False

Filters are processed in the order they are registered and in an "exit early" philosophy, none after are called if any decide to override the transmission state of an entity. Although we are looping through all entities every time, it is very fast because all it does in most cases is comparing a few bits and move on. At least, depending of the precision of your filters. In any cases, it is much more efficient than any other hook since we are only processing once per frame per player as opposed to once per frame, per player, as well as per entity, etc.

There are two rules; the world is always transmitted and the player are always transmitted to themselves. If we don't, the clients are crashing randomly.

Anyways, here are some Windows builds if anyone wants to test: https://drive.google.com/drive/folders/1Tls07h9H_UTjflwV9hP3iEcN-4OFCDei?usp=sharing

[vinci6k]

Oh my.. This is amazing! I'm currently out of town so I'm unable to play around with it. Can't wait to get my hands on this.

[CookStar]

After seeing vinci6k's comment and jordanbriere's commit, I came up with an implementation that merges both of them.
This commit is for the core-part (C++). CookStar@d739f6e

Sure, this is still in the draft stage, but I think it will be easy to use and fast.
And, if we implement And and Or in SIMD, we can furthermore speed up the process.

Example:

from events import Event
from players.entity import Player

from entities.transmit import transmit_manager

@Event("player_say")
def on_player_say(game_event):
    player = Player.from_userid(game_event["userid"])
    entity = player.view_entity

    if game_event["text"] == "hide":
        transmit_manager.hide(entity.index)
    if game_event["text"] == "hide_from":
        transmit_manager.hide_from(entity.index, player.index)

    if game_event["text"] == "unhide":
        transmit_manager.unhide(entity.index)
    if game_event["text"] == "unhide_from":
        transmit_manager.unhide_from(entity.index, player.index)

    if game_event["text"] == "is_hidden":
        #Print if the entity is hidden globally.
        print(transmit_manager.is_hidden(entity.index))
    if game_event["text"] == "is_hidden_from":
        #Print if the entity is hidden to a specific player
        print(transmit_manager.is_hidden_from(entity.index, player.index))
    if game_event["text"] == "get_hidden_states":
        #Get a list of player indexes where the entity is hidden.
        print(transmit_manager.get_hidden_states(entity.index))

    if game_event["text"] == "reset":
        transmit_manager.reset(entity.index)
    if game_event["text"] == "reset_from":
        transmit_manager.reset_from(entity.index, player.index)
    if game_event["text"] == "reset_all":
        transmit_manager.reset_all()

I completely discarded the Python callbacks in this implementation. I don't think we need a conditional lookup in callbacks to hide an entity.

Right now the filters are consuming a lot of memory, so I wonder if it would be a good idea to increase or decrease the filters on player active and disconnect to reduce memory consumption.

[jordanbriere]

And, if we implement And and Or in SIMD, we can furthermore speed up the process.

It could also be optimized by looping yourself instead of relying on And and Or. This would not only cut the amount of iterations by half, but would also remove the need for patching the SDK (which should really be a last resort).

Right now the filters are consuming a lot of memory, so I wonder if it would be a good idea to increase or decrease the filters on player active and disconnect to reduce memory consumption.

You are using a lot of memory because you are using the absolute max that is in bits resulting in your vector being 4 to 8 times bigger than what you really need depending of the game. Basically, your vector has entries for 255 players, while the max is 32 on most games, 64 on CS games, 4 on L4D, etc.

That said, your implementation doesn't seem to take into consideration that indexes can be re-used, or that players can join after an entity was globally hidden so you definitely should listen for (at least) player connection and entity deletion regardless.

I completely discarded the Python callbacks in this implementation. I don't think we need a conditional lookup in callbacks to hide an entity.

I tend to disagree here. I originally pitched that hide/unhide design but I grew against it because it simply can't compare with the flexibility a dynamic filtering system has to offers. For example, let's say I want to hide teammates from seeing each other. Using a filter, I could just do the following:

from entities.transmit import TransmitCriteria
from entities.transmit import TransmitFilter

@TransmitFilter(criteria=TransmitCriteria.is_player)
def transmit_filter(entity, player):
    return entity.team_index == player.team_index

While with your implementation, I would have to listen for team changes, player connection, disconnection, etc. in order to loop through all players every time and ensure their state are constantly up-to-date. Yet, this is a very basic example and could get even more trickier quickly if I wanted for example, not only hide entities from a specific team, but also based on their location on the map, etc. I honestly don't believe the rare few cases I can think of where simply having global/per-player switches would be viable as opposed to register my own hook.

Anyways, I have a clear vision of where I want to take my implementation but just haven't gotten to it yet and probably won't have any time to allocate for a second pass for at least another two weeks minimum. There are quite many flaws in my first pass I want to address, as well as minimizing the syntax to both optimize and improve flexibility. Let's discuss the pros and cons then I guess.