Error loading SP

[waldotf]

This is almost certainly a system configuration issue on my end, but I'm getting the following when I try to load SP:

[Source.Python] Loading...
[Source.Python]
[SP]
  File '../addons/source-python/packages/source-python/__init__.py', line 190: UserWarning
    BaseEntityOutput.fire_output not found. OnEntityOutput listener will not fire.

[Source.Python]
[SP]
  File '../addons/source-python/packages/source-python/__init__.py', line 417: UserWarning
    No global pointer found for "GlobalEntityList".

[Source.Python] Loaded successfully.

As you'd expect, this causes many plugin features to misbehave/segfault/etc. Guessing this is a system config issue, though I do have the required libs installed. Here's sp info:

[Source.Python]
IMPORTANT: Please copy the full output.
--------------------------------------------------------
Checksum      : e22a327d8f387ea42b03cc4081156ecd
Date          : 2023-02-14 02:39:44.143722
OS            : Linux-5.10.0-10-amd64-x86_64-with-debian-10.13
Game          : tf2
SP version    : 719
Github commit : 0171b36297c23d480c17631561c98c0ff2afca28
Server plugins:
   00: Metamod:Source 1.11.0-dev+1148
   01: Source.Python, (C) 2012-2021, Source.Python Team.
SP plugins:
   00: tempus_loader
   01: noshake
--------------------------------------------------------

Separately, the registration process for the forums seems to be broken (no activation emails get sent). I retried a few times over a day or so.

[jordanbriere]

I'm getting the following when I try to load SP

What do you get when launching with +developer 1 as well as having your logging level (into ../cfg/source-python/core_settings.ini) set to 5? Also, what is the output of version?

[waldotf]

With `+developer 1` and `level = 5`:

[Source.Python] Loading...
[Source.Python] Retrieving game directory...
[Source.Python] Game directory: /srv/srcds/tf
[Source.Python] Checking if update stage 2 can be applied... No.
[Source.Python] Loading library: Python3/plat-linux/libpython3.6m.so.1.0
[Source.Python] Loading library: bin/core.so
[Source.Python] Connecting tier1 libraries...
[Source.Python] Connecting tier2 libraries...
[Source.Python] Retrieving engine interfaces...
[Source.Python] Interface VEngineServer023 at f674148c
[Source.Python] Interface GAMEEVENTSMANAGER002 at f67919a0
[Source.Python] Interface ISERVERPLUGINHELPERS001 at f67d3340
[Source.Python] Interface EngineTraceServer003 at f67916e0
[Source.Python] Interface IEngineSoundServer003 at f6791610
[Source.Python] Interface VEngineRandom001 at f6741190
[Source.Python] Interface VFileSystem022 at f7590d40
[Source.Python] Interface VEngineServerStringTable001 at f679dba0
[Source.Python] Interface VPhysics031 at f32efde0
[Source.Python] Interface VPhysicsCollision007 at f32efec0
[Source.Python] Interface VPhysicsSurfaceProps001 at f32f03a0
[Source.Python] Interface MDLCache004 at f349b3a0
[Source.Python] Interface VoiceServer002 at f6741494
[Source.Python] Retrieving game interfaces...
[Source.Python] Interface PlayerInfoManager002 at f16d40c0
[Source.Python] Interface BotManager001 at f16d40bc
[Source.Python] Interface ServerGameDLL012 at f17988a4
[Source.Python] Interface ServerGameClients004 at f16c2020
[Source.Python] Interface VSERVERTOOLS003 at f16b9d6c
[Source.Python] Retrieving global variables...
[Source.Python] Initializing mathlib...
[Source.Python] Initializing server and say commands...
[Source.Python] Registering say and say_team commands...
[Source.Python] Registering ConVar accessor...
[Source.Python] Initializing python...
[Source.Python] Python home path set to /srv/srcds/tf/addons/source-python/Python3
[Source.Python] sys.executable set to /srv/srcds/tf/addons/source-python/Python3/plat-linux/libpython3.6m.so.1.0
[Source.Python] Python version 3.6.1 (default, Jun 16 2017, 09:50:37)
[GCC 4.8.4] initialized!
[Source.Python] Setting sys.argv...
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/source-python to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/Python3/plat-linux to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/Python3/lib-dynload to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/site-packages to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/custom to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/plugins to path
[Source.Python] Initializing _core._cache module
[Source.Python] Initializing _memory module
[Source.Python] Initializing _bitbuffers module
[Source.Python] Initializing _colors module
[Source.Python] Initializing _commands._client module
[Source.Python] Initializing _commands module
[Source.Python] Initializing _commands._say module
[Source.Python] Initializing _commands._server module
[Source.Python] Initializing _cvars module
[Source.Python] Initializing _effects module
[Source.Python] Initializing _effects._base module
[Source.Python] Initializing _engines module
[Source.Python] Initializing _engines._server module
[Source.Python] Initializing _engines._sound module
[Source.Python] Initializing _engines._trace module
[Source.Python] Initializing _engines._gamerules module
[Source.Python] Initializing _entities._constants module
[Source.Python] Initializing _entities._helpers module
[Source.Python] Initializing _entities module
[Source.Python] Initializing _entities._factories module
[Source.Python] Initializing _entities._datamaps module
[Source.Python] Initializing _entities._props module
[Source.Python] Initializing _entities._entity module
[Source.Python] Initializing _core module
[Source.Python] Initializing _events module
[Source.Python] Initializing _globals module
[Source.Python] Initializing _keyvalues module
[Source.Python] Initializing _listeners module
[Source.Python] Initializing _mathlib module
[Source.Python] Initializing _messages module
[Source.Python] Initializing _net_channel module
[Source.Python] Initializing _physics module
[Source.Python] Initializing _players._bots module
[Source.Python] Initializing _players._constants module
[Source.Python] Initializing _players._helpers module
[Source.Python] Initializing _players module
[Source.Python] Initializing _players._voice module
[Source.Python] Initializing _plugins module
[Source.Python] Initializing _filters._recipients module
[Source.Python] Initializing _filesystem module
[Source.Python] Initializing _steam module
[Source.Python] Initializing _stringtables module
[Source.Python] Initializing _studio module
[Source.Python] Initializing _studio._constants module
[Source.Python] Initializing _studio._cache module
[Source.Python] Initializing _weapons._constants module
[Source.Python] Initializing _weapons._scripts module
[Source.Python] Initializing _weapons._entity module
[Source.Python] Loading main module...
[Source.Python] Setting up exception hooks...
[Source.Python] Setting up data update...
[Source.Python] Checking for data updates...
[Source.Python] No new data is available.
[Source.Python] Setting up translations...
[Source.Python] Setting up data...
[Source.Python]
[SP]
  File '../addons/source-python/packages/source-python/__init__.py', line 190: UserWarning
    BaseEntityOutput.fire_output not found. OnEntityOutput listener will not fire.

[Source.Python] Setting up global pointers...
[Source.Python] Setting up global "server" variables...
[Source.Python] Setting up global "factory_dictionary" variables...
[Source.Python] Setting up the "sp" command...
[Source.Python] OnLevelEnd.__init__<<function _level_end at 0xe9b7680c>>
[Source.Python] OnLevelEnd.__init__ - Registering
[Source.Python] Setting up auth...
[Source.Python] Setting up user settings...
[Source.Python] OnClientDisconnect.__init__<<function on_player_disconnect at 0xe9b35974>>
[Source.Python] OnClientDisconnect.__init__ - Registering
[Source.Python] Repeat.__init__: <<function _radio_refresh at 0xe9b3e2b4>> <()> <{}>
[Source.Python] Repeat.__init__: <<function _esc_refresh at 0xe9b3e2fc>> <()> <{}>
[Source.Python] OnClientDisconnect.__init__<<function on_player_disconnect at 0xe9b3e3d4>>
[Source.Python] OnClientDisconnect.__init__ - Registering
[Source.Python] Setting up entities listener...
[Source.Python]
[SP]
  File '../addons/source-python/packages/source-python/__init__.py', line 417: UserWarning
    No global pointer found for "GlobalEntityList".

[Source.Python] Setting up versioning...
[Source.Python] Pre-loading libsqlite3.so.0...
[Source.Python] Retrieving the current cache notifier...
[Source.Python] Setting the new cache notifier...
[Source.Python] Loaded successfully.

`version`:

Build Label:           7757534   # Uniquely identifies each build
Network PatchVersion:  7757534   # Determines client and server compatibility
Protocol version:           24   # High level network protocol version
Server version:        7757534
Server AppID:           232250

[jordanbriere]

Your ../addons/source-python/data/source-python/* directory appears to be missing entirely. My first guess, would be that the data update failed somehow (or, at the very least, its extraction didn't succeed). Try to delete your ../addons/source-python/data/source-python-data.zip file and restart your server.

[waldotf]

That seems to be it; here's the first boot after deleting the .zip:

log

[Source.Python] Loading...
[Source.Python] Retrieving game directory...
[Source.Python] Game directory: /srv/srcds/tf
[Source.Python] Checking if update stage 2 can be applied... No.
[Source.Python] Loading library: Python3/plat-linux/libpython3.6m.so.1.0
[Source.Python] Loading library: bin/core.so
[Source.Python] Connecting tier1 libraries...
[Source.Python] Connecting tier2 libraries...
[Source.Python] Retrieving engine interfaces...
[Source.Python] Interface VEngineServer023 at f66e448c
[Source.Python] Interface GAMEEVENTSMANAGER002 at f67349a0
[Source.Python] Interface ISERVERPLUGINHELPERS001 at f6776340
[Source.Python] Interface EngineTraceServer003 at f67346e0
[Source.Python] Interface IEngineSoundServer003 at f6734610
[Source.Python] Interface VEngineRandom001 at f66e4190
[Source.Python] Interface VFileSystem022 at f7559d40
[Source.Python] Interface VEngineServerStringTable001 at f6740ba0
[Source.Python] Interface VPhysics031 at f31efde0
[Source.Python] Interface VPhysicsCollision007 at f31efec0
[Source.Python] Interface VPhysicsSurfaceProps001 at f31f03a0
[Source.Python] Interface MDLCache004 at f39d23a0
[Source.Python] Interface VoiceServer002 at f66e4494
[Source.Python] Retrieving game interfaces...
[Source.Python] Interface PlayerInfoManager002 at f16860c0
[Source.Python] Interface BotManager001 at f16860bc
[Source.Python] Interface ServerGameDLL012 at f174a8a4
[Source.Python] Interface ServerGameClients004 at f1674020
[Source.Python] Interface VSERVERTOOLS003 at f166bd6c
[Source.Python] Retrieving global variables...
[Source.Python] Initializing mathlib...
[Source.Python] Initializing server and say commands...
[Source.Python] Registering say and say_team commands...
[Source.Python] Registering ConVar accessor...
[Source.Python] Initializing python...
[Source.Python] Python home path set to /srv/srcds/tf/addons/source-python/Python3
[Source.Python] sys.executable set to /srv/srcds/tf/addons/source-python/Python3/plat-linux/libpython3.6m.so.1.0
[Source.Python] Python version 3.6.1 (default, Jun 16 2017, 09:50:37)
[GCC 4.8.4] initialized!
[Source.Python] Setting sys.argv...
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/source-python to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/Python3/plat-linux to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/Python3/lib-dynload to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/site-packages to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/packages/custom to path
[Source.Python] Adding /srv/srcds/tf/addons/source-python/plugins to path
[Source.Python] Initializing _core._cache module
[Source.Python] Initializing _memory module
[Source.Python] Initializing _bitbuffers module
[Source.Python] Initializing _colors module
[Source.Python] Initializing _commands._client module
[Source.Python] Initializing _commands module
[Source.Python] Initializing _commands._say module
[Source.Python] Initializing _commands._server module
[Source.Python] Initializing _cvars module
[Source.Python] Initializing _effects module
[Source.Python] Initializing _effects._base module
[Source.Python] Initializing _engines module
[Source.Python] Initializing _engines._server module
[Source.Python] Initializing _engines._sound module
[Source.Python] Initializing _engines._trace module
[Source.Python] Initializing _engines._gamerules module
[Source.Python] Initializing _entities._constants module
[Source.Python] Initializing _entities._helpers module
[Source.Python] Initializing _entities module
[Source.Python] Initializing _entities._factories module
[Source.Python] Initializing _entities._datamaps module
[Source.Python] Initializing _entities._props module
[Source.Python] Initializing _entities._entity module
[Source.Python] Initializing _core module
[Source.Python] Initializing _events module
[Source.Python] Initializing _globals module
[Source.Python] Initializing _keyvalues module
[Source.Python] Initializing _listeners module
[Source.Python] Initializing _mathlib module
[Source.Python] Initializing _messages module
[Source.Python] Initializing _net_channel module
[Source.Python] Initializing _physics module
[Source.Python] Initializing _players._bots module
[Source.Python] Initializing _players._constants module
[Source.Python] Initializing _players._helpers module
[Source.Python] Initializing _players module
[Source.Python] Initializing _players._voice module
[Source.Python] Initializing _plugins module
[Source.Python] Initializing _filters._recipients module
[Source.Python] Initializing _filesystem module
[Source.Python] Initializing _steam module
[Source.Python] Initializing _stringtables module
[Source.Python] Initializing _studio module
[Source.Python] Initializing _studio._constants module
[Source.Python] Initializing _studio._cache module
[Source.Python] Initializing _weapons._constants module
[Source.Python] Initializing _weapons._scripts module
[Source.Python] Initializing _weapons._entity module
[Source.Python] Loading main module...
[Source.Python] Setting up exception hooks...
[Source.Python] Setting up data update...
[Source.Python] Checking for data updates...
[Source.Python] New data is available. Downloading...
[Source.Python] Downloading file (http://data.sourcepython.com/source-python-data.zip) to /srv/srcds/tf/addons/source-python/data/source-python-data.zip ...
[Source.Python] File has been downloaded. Time elapsed: 0.74 seconds
[Source.Python] Removing /srv/srcds/tf/addons/source-python/data/source-python ...
[Source.Python] Extracting data in /srv/srcds/tf/addons/source-python/data ...
Segmentation fault
cat: hlds.7.pid: No such file or directory
email debug.log to linux@valvesoftware.com
Tue Feb 14 06:32:47 UTC 2023: Server Quit

Another restart looks the same as the original log. Not sure if it's what matters here, but here's the installed zlib:

root@71a0d0effe7a:/# dpkg -l | grep zlib
ii  zlib1g:i386               1:1.2.11.dfsg-1+deb10u2     i386         compression library - runtime

[jordanbriere]

Another restart looks the same as the original log.

That's because SP assumes the data is up-to-date since the zip's checksum is the same. A quick google search for cat: hlds.7.pid: No such file or directory yielded the following thread as first result: tf2 crashing 'cat: hlds.13915.pid: No such file or directory'. Perhaps it can points you in the right direction.

[Ayuto]

Maybe the server is unable to extract the new data, because of a permission problem?

[waldotf]

I tried a few extra library packages suggested in various posts without success. All of srcds/ is owned by the user which is running the server process.

I pared out _unpack_data from

Source.Python/addons/source-python/packages/source-python/core/update.py

Line 311 in 1f87696

def _unpack_data(path):

into a simple test.py:

from zipfile import ZipFile

def _unpack_data(path):
    """Unpack ``source-python-data.zip`` into the given path.
    :param Path path:
        The path the data file should be unpacked into.
    """
    with ZipFile("source-python-data.zip") as zip:
        zip.extractall(path)

_unpack_data("source-python-data-test")

and that extracts fine as the same user, using the system Python (3.7.3). If I put that in tf/addons/source-python/data/source-python and restart, everything works as expected (i.e. this seems to be the only issue).

Since it's getting to this log line

Source.Python/addons/source-python/packages/source-python/core/update.py

Line 317 in 1f87696

update_logger.log_debug(f'Extracting data in {path} ...')

and failing to extract, my guess is it's something specific to SP's Python install. Is there an easy way to invoke that?

[waldotf]

Looks like Zipfile isn't threadsafe in our version (fix only backported as far as 3.7). Not sure if that's relevant at all.

The crash is definitely isolated to that call, but I can't reproduce it outside of SP and I don't know how to get a proper backtrace for it with gdb because of all the forking in srcds.

[jordanbriere]

Looks like Zipfile isn't threadsafe in our version (fix only backported as far as 3.7). Not sure if that's relevant at all.

Very unlikely, because the extraction is done on the main thread.

The crash is definitely isolated to that call, but I can't reproduce it outside of SP and I don't know how to get a proper backtrace for it with gdb because of all the forking in srcds.

If you look at the debug.log generated, it should contain the entire stacktrace. I suspect one of the call is not resolved from the right shared library likely due to a mis-configurated multiarch or something. More than likely zlib (or one of its dependencies). Try to run ldd on addons/source-python/Python3/lib-dynload/zlib.cpython-36m-i386-linux-gnu.so.

[waldotf]

srcds/debug.log is just entries that look like this:

----------------------------------------------
CRASH: Wed Feb 15 06:12:54 UTC 2023
Start Line: ./srcds_linux -game tf -debug -port 27025 +map jump_beef -maxplayers 24 +developer 1 -norestart
End of Source crash report
----------------------------------------------

Doesn't look like any of these are the wrong arch, but I don't know what the missing libpython means.

# ldd tf/addons/source-python/Python3/lib-dynload/zlib.cpython-36m-i386-linux-gnu.so
        linux-gate.so.1 (0xf7fc6000)
        libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xf7f97000)
        libpython3.6m.so.1.0 => not found
        libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7f76000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7d98000)
        /lib/ld-linux.so.2 (0xf7fc8000)

[jordanbriere]

Actually, I think I know exactly why it happens. Try the following:

• Download and install: [removed attachment]
• Prior to launching your server export the following (make sure the paths are correct, of course):

export LD_PRELOAD=/addons/source-python/Python3/plat-linux/libz.so.1
srcds_run -game tf ...

[waldotf]

Same segfault with the preloaded zlib.

[jordanbriere]

Try: [removed attachment]

[waldotf]

Oops, looks like I was invoking srcds_run weirdly and that prevented it from properly preloading -- using either zlib 1.2.3 or 1.2.5 fixes the issue.

The system (Debian 10) version was 1.2.11, which didn't work. Debian 9 repos (out of support since 2020-07-06) have 1.2.8, which did work.

Thanks a ton for the help.

[jordanbriere]

Theoretically, pre-loading your installed version (/lib/i386-linux-gnu/libz.so.1) should do the trick as well, but you may face different issues in different contexts. Actually, for science, could you test it out? I don't currently have time to go in details, but I will explain what is going on and why it happens in a few hours.