high cpu load after upgrade 7.0.4 to 8.0.2 and lucene disabled but still present in the logs

[seal20]

Steps to reproduce

I don't really know how to reproduce it but it keeps happening... What I suppose

1. Upgrade to 8.0.2
2. Lucene start indexing or optimizing resulting in very high cpu load
3. Disable lucene
4. the high cpu load keep happening and the log show error related to lucene

Expected behaviour

no high cpu load and once lucence is disabled it doesn't do anything

Actual behaviour

high cpu load and still lucene errors in the log

Server configuration

Operating system: Centos 6.6 (virtuozzo VPS)

**Web server:**Apache/2.4.10 (Unix)

**Database:**MySQL Ver 14.14 Distrib 5.5.42, for Linux (x86_64) using readline 5.1

**PHP version:**PHP 5.4.35 (cli) (built: Nov 29 2014 08:31:07)
Zend Engine v2.4.0, eAccelerator v0.9.6.1-ea, ionCube PHP Loader v4.6.1, Zend Guard Loader v3.3,

ownCloud version: (see ownCloud admin page)
8.0.2.0

Updated from an older ownCloud or fresh install:
yes from 7.0.4

List of activated apps:

Enabled:
 - activity
 - calendar
 - contacts
 - documents
 - files
 - files_pdfviewer
 - files_sharing
 - files_texteditor
 - files_trashbin
 - files_versions
 - files_videoviewer
 - firstrunwizard
 - gallery
 - news
 - ownnote
 - tasks
 - templateeditor
 - updater
Disabled:
 - external
 - files_encryption
 - files_external
 - files_locking
 - provisioning_api
 - search_lucene
 - user_external
 - user_ldap
 - user_webdavauth

The content of config/config.php:

$CONFIG = array (
  'instanceid' => 'oc3053678514',
  'passwordsalt' => '',
  'datadirectory' => '/path/oc_data',
  'dbtype' => 'mysql',
  'version' => '8.0.2.0',
  'dbname' => '',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'dbuser' => '',
  'dbpassword' => '',
  'installed' => true,
  'forcessl' => true,
  'loglevel' => 0,
  'log_rotate' => '104857600',
  'theme' => '',
  'maintenance' => false,
  'trusted_domains' => 
  array (
    0 => '.com',
  ),
  'mail_smtpmode' => 'smtp',
  'mail_from_address' => 'notifications',
  'mail_domain' => '.com',
  'overwrite.cli.url' => '/owncloud',
  'allow_user_to_change_display_name' => true,
  'logtimezone' => 'Asia/Tokyo',
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'preview_office_cl_parameters' => ' --headless --nologo --nofirststartwizard --invisible --norestore -convert-to pdf -outdir ',
  'secret' => ',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '465',
  'appcodechecker' => false,
  'custom_csp_policy' => 'default-src \'self\'; script-src \'self\' \'unsafe-inline\' \'unsafe-eval\'; style-src \'self\' \'unsafe-inline\'; frame-src *; img-src *; font-src \'self\' data:; media-src *',
); 

The last custom_csp_policy is required by ownnotes

Are you using external storage, if yes which one: local/smb/sftp/...
no

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

not relevant as it happens when nobody os connected, perhaps cron.php?

Browser:

Operating system:

Logs

Web server error log

[Thu Mar 26 04:21:43.892165 2015] [authz_core:error] [pid 7066] [client 91.121.100.66:41561] AH01630: client denied by server configuration: /home/agoremix/public_html/.htaccess
[Thu Mar 26 05:00:03.454043 2015] [core:crit] [pid 7065] (13)Permission denied: [client 184.105.139.67:37796] AH00529: /home/agoremix/public_html/oc_data/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/home/agoremix/public_html/oc_data/' is executable
[Thu Mar 26 05:00:03.454104 2015] [authz_core:error] [pid 7065] [client 184.105.139.67:37796] AH01630: client denied by server configuration: /home/agoremix/public_html/.htaccess
[Thu Mar 26 05:08:16.855082 2015] [mpm_prefork:notice] [pid 20558] AH00171: Graceful restart requested, doing restart
[Thu Mar 26 05:08:16.938089 2015] [auth_digest:notice] [pid 20558] AH01757: generating secret for digest authentication ...
[Thu Mar 26 05:08:17.026797 2015] [ssl:warn] [pid 20558] AH01906: server.domain.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Mar 26 05:08:17.030115 2015] [ssl:warn] [pid 20558] AH01906: books.domain.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Mar 26 05:08:17.063531 2015] [mpm_prefork:notice] [pid 20558] AH00163: Apache/2.4.10 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4 configured -- resuming normal operations
[Thu Mar 26 05:08:17.063547 2015] [core:notice] [pid 20558] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
[Thu Mar 26 07:03:00.624653 2015] [mpm_prefork:notice] [pid 20558] AH00169: caught SIGTERM, shutting down

ownCloud log (data/owncloud.log)

{"reqId":"1b4141ed99b35f76483d6381c1312d0b","remoteAddr":"","app":"PHP","message":"Undefined index: REQUEST_URI at \/home\/domain\/public_html\/owncloud\/apps\/calendar\/appinfo\/app.php#41","level":3,"time":"2015-03-26T22:48:13+09:00","method":"--","url":"--"}
{"reqId":"da309fba08bfab2be71b7285a9602f8a","remoteAddr":"","app":"PHP","message":"Undefined index: REQUEST_URI at \/home\/domain\/public_html\/owncloud\/apps\/calendar\/appinfo\/app.php#41","level":3,"time":"2015-03-26T22:48:50+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"PHP","message":"Undefined index: REQUEST_URI at \/home\/domain\/public_html\/owncloud\/apps\/calendar\/appinfo\/app.php#41","level":3,"time":"2015-03-26T23:00:02+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"news","message":"Configuration value \"simplePieCacheDuration\" does not exist. Ignored value.","level":2,"time":"2015-03-26T23:00:02+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"background job optimizing index for sylvain","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"removed 0 files from index","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"background job optimizing index for roger","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"removed 0 files from index","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"background job optimizing index for chouette","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}
{"reqId":"458ec71b2ff62ca44657fdfc03327de2","remoteAddr":"","app":"search_lucene","message":"removed 0 files from index","level":0,"time":"2015-03-26T23:00:22+09:00","method":"--","url":"--"}

Problem I don't know exactely at what time the high load happened but I received a high load abuse warning everyday for the last three days and today was the LAST warning... but I couldn't witness myself the high load. If I run the cron job with lucene, there was a high load so I disabled lucene and the cron.php soon finish without high load. However the log still show lucene error and sometimes in the day the high load happens again.
I disabled apache and cron jobs for the moment but I want to find a way to debug this without getting kicked out by my provider.

[nickvergessen]

@seal20 please change your email/smtp password as it has been public.
I removed the entries in your report, but you should still change them as a precaution.

[nickvergessen]

Also I think https://github.com/owncloud/search_lucene might be a more suitable place

[seal20]

@nickvergessen Thanks I changed the smtp password, stupid mistakes.

I was wondering which repo to submit the issue but because I think the app doesn't get "disabled" I though it would be suitable to submit it in core. Will move it if you really think it is more suitable.

[seal20]

I could "fix" the high load issue... by disabling some other non-owncloud background jobs.
However, I can confirm that even if lucene is disabled it still log errors and lucene indexing is never ending when enabled using a lot of cpu for a very long time. The "count down of files remaining" starts at around 140K, goes fast in the begining and stall at 139K after 1 hour, then I killed the php and disabled lucene.
So I would like to separate this issue in 2, please advise if they better fit in the lucene repo:

1. Lucene even if disabled still logs error so is still doing something
2. Indexing in lucene use too much cpu and takes too long (duplicate of Search keeps on indexing owncloud-archive/search_lucene#44 ? ; definitevely a lucene issue)

[VicDeo]

@seal20 please have a look at owncloud-archive/search_lucene#84 for the 1st point