Missing posts from the index

Hi @saumya010

To be certain, you’ve logged in to your algolia.com dashboard and verified that these posts are listed there, and when performing a query using that UI, the posts in question are among the results? However, when making the same query from your WP install, the posts in question are NOT showing?

Have you done any customization or perhaps filter configuration on the WordPress side that could potentially remove them from results?

We don’t have much for logs in the current version but know we have a GitHub issue around the topic.

Extra to be certain, you haven’t made use of either of these filters, correct?

• algolia_should_index_searchable_post
• algolia_should_index_post

By default as long as a given post is published, and not password protected, it should be getting indexed https://github.com/WebDevStudios/wp-search-with-algolia/blob/2.8.3/includes/indices/class-algolia-searchable-posts-index.php#L95-L109 but as you’ll note, there’s one of the filters above. For brevity I am not linking to the other one.

As an extra test, have you edited the specific post and just clicked “update”, and then confirm whether or not it’s now in the index again?

That’s going to be a troubleshooting topic on your end with the specific install in question. The plugin isn’t coded to potentially randomly de-index content unless it’s been trashed/deleted in WordPress.

The only way I could think of **potential** fail to re-index, would be if there’s somehow an error during the process of indexing. As is, the plugin deletes the entire record right before re-pushing the entire new state of the record, as opposed to patch/updating just the parts of it. For example if you just renamed the post, and clicked update, the plugin removes, and processes and re-indexes the whole post, instead of sending just the new title value. If the last part of that is somehow failing…I could see potential in “weird deletion” but that’s the only way off the top of my head.

Are you doing any sort of routine bulk re-indexing? Perhaps as part of a daily schedule or similar?

All of the post watching happens in https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/watchers/class-algolia-post-changes-watcher.php

This is away from bulk re-indexing, and just at the per-post level.

We still don’t have all that much for meaningful logging outside of spots that attempt to log thrown exceptions.

While I know we’re getting close to a new release, we’re not there yet and it won’t be in the next week or so. I bring this up because there’s room where I would not fault you for self-modifying the plugin for troubleshooting purposes to try and track things down a bit more manually.

We do indeed run deleteObjects() note the plurality, in say https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/indices/class-algolia-searchable-posts-index.php#L478-L497 and that I do believe gets mapped to deleteObject() singular, within Algolia’s PHP client that we ship with the plugin. This has been true from the point that we forked the plugin originally, and how Algolia themselves did this spot before us. Not pointing blame anywhere, just providing context.

Best I can tell, this was done with the intent of just “remove the post completely, and push everything fresh”, instead of trying to determine all of the matches and the differences between the currently indexed content and the to-be-indexed content. More efficient to delete previous and push current.

The question ultimately ends up being why isn’t some of the content not making it successfully through an update_records() method, which is getting invoked right after the deletion.

Sorry for throwing some code your way, but I’m hopeful you can generally comprehend it.

Here’s the update_records() method location:

https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/indices/class-algolia-index.php#L334-L384

Cases where it may return early, or indicate issues:

1. if already re-indexing things elsewhere.
2. if $this->sanitize_json_data() returns an empty dataset or throws an exception.
3. If the $index->saveObjects method throws an exception. This one is going to be from Algolia’s PHP client as well.

I’m curious if we should wrap parent::update_records( $post, $records ); from update_post_records() in some sort of try/catch block, or something, to try and better indicate a failure. If error_log() is working for your server, then the logs may have details around potential exceptions.

@blgerber The plugin isn’t going to automatically index everything all at once, once activated and the application details filled in. You’ll want to make use of the “bulk index” functionality to have WP push all of your eligible content indexed.

The indexes will self manage, by say saving a given post, and that one post will re-index. However, without bulk indexing, you’d need to click to edit and save EACH post.

Also just to ensure it’s known, saving the logic for what to maybe index, also benefits from a bulk index run to get just what newly should be indexed, into the index.