[enhancement](meta) Sync tablet meta even if local state is not runni…

…ng (#45821)

Avoid a running tablet keep not running state on single one BE with no
query. Even if it is a tablet with high compaction score, compaction
will fail on this BE since not running state.

Before this PR, scheduled tablet meta sync will skip not running
tablets. In this PR, we include those tablets in meta sync procedure to
avoid long-term inaccurate tablet state.

be/src/cloud/cloud_tablet.cpp

@@ -873,9 +873,6 @@ Status CloudTablet::sync_meta() {
         }
         return st;
     }
-    if (tablet_meta->tablet_state() != TABLET_RUNNING) { // impossible
-        return Status::InternalError("invalid tablet state. tablet_id={}", tablet_id());
-    }
 
     auto new_ttl_seconds = tablet_meta->ttl_seconds();
     if (_tablet_meta->ttl_seconds() != new_ttl_seconds) {

be/src/cloud/cloud_tablet_mgr.cpp

@@ -261,9 +261,6 @@ void CloudTabletMgr::sync_tablets(const CountDownLatch& stop_latch) {
 
     for (auto& weak_tablet : weak_tablets) {
         if (auto tablet = weak_tablet.lock()) {
-            if (tablet->tablet_state() != TABLET_RUNNING) {
-                continue;
-            }
             int64_t last_sync_time = tablet->last_sync_time_s;
             if (last_sync_time <= last_sync_time_bound) {
                 sync_time_tablet_set.emplace(last_sync_time, weak_tablet);