Cedar Standalone with XCode 8/ios-sim 5.0.8

[mkanthan]

Before my massive upgrade, I was running XCode 7.3.1, and ios-sim 3.1.1, with Cedar 0.13.1 and all was working fine. I have Cedar set up in the "standalone" configuration and have a rake task which executes the xcodebuild command and runs the Specs in ios-sim. Running the Specs target from XCode works as well. I used the Cedar Rakefile from the templates with some modifications:

...
desc "Run Specs"
task :Specs => [:check_ios_sim_availability, :check_ios_sim_version, :build_Specs] do

  env_vars = {
    "CEDAR_REPORTER_CLASS" => CedarTargetSpecs.reporter_class,
    "CEDAR_REPORTER_OPTS" => "nested",
    "CEDAR_RANDOM_SEED" => "0",
    "OS_ACTIVITY_MODE" => "disable"
  }

  CedarTargetSpecs.with_env_vars(env_vars) do
    env_var_options = ""
    env_vars.each { |k,v| env_var_options << "--setenv " << k << "=" << v << " "  }
    CedarTargetSpecs.system_or_exit "ios-sim launch #{File.join(CedarTargetSpecs.target_build_dir, "#{CedarTargetSpecs::UI_SPECS_TARGET_NAME}.app")} #{env_var_options} --devicetypeid 'com.apple.CoreSimulator.SimDeviceType.iPad-Air-2, 10.0' | tee /dev/stderr | grep -q ', 0 failures'";
  end
end

I won't post the whole Rakefile here because I believe it's irrelevant.

Enter XCode 8/Cedar 1.0. Upon upgrading and running my rake Specs command - when it executes the ios-sim launch ... command, I'm greeted with an error:

Failed to find class DTiPhoneSimulatorSystemRoot at runtime.

After digging through ios-sim's repository, I learned that the DTiPhoneSimulatorSystemRoot is not present in the newest XCode, prompting me to upgrade (no longer via brew - you now have to install ios-sim via npm). I've upgraded to ios-sim 5.0.8. Upon running rake Specs, it seems to run, but the task no longer completes, and I no longer get the same test output. Instead, I get most of XCode's warnings and such:

Oct 13 19:56:00 --- last message repeated 69 times ---
Oct 13 19:56:00 My-MBP Specs[65978]: Warning: Attempt to present <LGSettingsViewController: 0x79dd24e0> on <LGViewController: 0x7e1cf5c0> whose view is not in the window hierarchy!
Oct 13 19:56:00 My-MBP Specs[65978]: Warning: Attempt to present <LGAboutScreenViewController: 0x7e171170> on <LGViewController: 0x79dd46b0> whose view is not in the window hierarchy!
Oct 13 19:56:00 My-MBP Specs[65978]: Warning: Attempt to present <LGHelpScreenViewController: 0x7e2640a0> on <LGViewController: 0x7e2622e0> whose view is not in the window hierarchy!
Oct 13 19:56:00 My-MBP assertiond[44474]: Could not create or rename power assertion for <BKNewProcess: 0x7ff6bb00dcb0; com.lingraphica.Specs; pid: 65978; hostpid: -1>: 0xe00002c9
Oct 13 19:56:04 --- last message repeated 25 times ---
Oct 13 19:56:04 My-MBP Specs[65978]: Warning: Attempt to present <UIAlertController: 0x7a2d1e00> on <LGViewController: 0x7e390600> whose view is not in the window hierarchy!
Oct 13 19:56:04 My-MBP Specs[65978]: Warning: Attempt to present <LGLoginViewController: 0x7e263920> on <LGViewController: 0x78f15ba0> whose view is not in the window hierarchy!
Oct 13 19:56:04 My-MBP Specs[65978]: Warning: Attempt to present <LGLoginViewController: 0x7e4b2ad0> on <LGViewController: 0x7e46f760> whose view is not in the window hierarchy!
Oct 13 19:56:04 My-MBP Specs[65978]: Warning: Attempt to present <LGLoginViewController: 0x7e284850> on <LGViewController: 0x78f2c820> whose view is not in the window hierarchy!
Oct 13 19:56:04 My-MBP assertiond[44474]: Could not create or rename power assertion for <BKNewProcess: 0x7ff6bb00dcb0; com.lingraphica.Specs; pid: 65978; hostpid: -1>: 0xe00002c9
Oct 13 19:56:05 --- last message repeated 5 times ---

Upon doing some more digging through ios-sim's Issues, it seems they changed the way they log the output, and it seems to be ignoring the printf statement in CDRDefaultReporter.m on line 97. Changing this to an NSLog gets me a tad further:

- (void)logText:(NSString *)linePartial {
    NSLog(@"%@", linePartial);
}

Now the output becomes:

....
Oct 13 20:10:03 My-MBP Specs[69607]: .
Oct 13 20:10:03 --- last message repeated 4 times ---
Oct 13 20:10:03 My-MBP Specs[69607]: PENDING LGLoginViewController webView:shouldStartLoadWithRequest:navigationType:navigationType: when the URL is for creating an account is pending 
Oct 13 20:10:03 My-MBP Specs[69607]: PENDING LGVideoAccess is pending 
Oct 13 20:10:03 My-MBP Specs[69607]: PENDING LGAppDelegate application:didFinishLaunchingWithOptions: should copy the database from the mainbundle to the documents directory 
Oct 13 20:10:03 My-MBP Specs[69607]: PENDING LGAppDelegate application:didFinishLaunchingWithOptions: should ask LGStoreKitFacade to observe purchase transactions 
Oct 13 20:10:03 My-MBP Specs[69607]: PENDING LGViewController when the URL scheme is 'video' is pending 
Oct 13 20:10:03 My-MBP Specs[69607]: 
    Finished in 39.5317 seconds
Oct 13 20:10:03 My-MBP Specs[69607]: 508 examples, 0 failures
Oct 13 20:10:03 My-MBP Specs[69607]: , 5 pending
$

So now, I get some partial test output and the task exits when it sees 0 failures printed in the log. Ideally, I'd want this to work on par with the way it used to, however, I'm not sure what the scope is. Is there any way to get this to print similar to the XCode console when running a standalone Specs target?

[tjarratt]

@mkanthan if I understand correctly, the problem you're facing is that it's difficult to get the original output printed by Cedar under Xcode 7 with Xcode 8 and the latest ios-sim?

Before, with Xcode 7 and ios-sim

Oct 13 20:10:03 My-MBP Specs[69607]: 508 examples, 0 failures, 5 pending

After, with Xcode 8

Oct 13 20:10:03 My-MBP Specs[69607]: 508 examples, 0 failures
Oct 13 20:10:03 My-MBP Specs[69607]: , 5 pending

That sounds mighty frustrating. I don't know how much we can do, as Cedar maintainers to help out here.

I personally, really truly dislike Test Bundles (for a variety of reasons), and I'm glad to hear you've been using standalone "spec suites" with Cedar. Unfortunately, this has historically meant trading more reasonable test execution for less well supported automated test running (with ios-sim and similar tools).

I don't think we'd want to change the call to printf to NSLog immediately, since it would end up making the default reporter output significantly messier in a lot of situations. It really seems like ios-sim needs to handle lines printed to stdout better. Perhaps there's some room for Cedar to print better, but I'm not entirely sure what that would be without more research.

[tjarratt]

I've also noticed that ios-sim is deprecated in favour of ioslib, which appears to be an npm package that exposes javascript functions that allow you to install and run apps on either a simulator or real device.

I haven't tried that out yet, but it looks promising.

[mkanthan]

It's actually very different with XCode 7 + ios-sim 3.1.1. It was nearly Identical to the XCode console output:

...
.      should return the sound files for instructions, prompt icons and every alphabet of higher difficulty spelling tasks which need the full keyboard
   LGVideoAccess
P    is pending


PENDING LGLoginViewController webView:shouldStartLoadWithRequest:navigationType:navigationType: when the URL is for creating an account is pending
PENDING LGAppDelegate application:didFinishLaunchingWithOptions: should copy the database from the mainbundle to the documents directory
PENDING LGAppDelegate application:didFinishLaunchingWithOptions: should ask LGStoreKitFacade to observe purchase transactions
PENDING LGViewController when the URL scheme is 'video' is pending
PENDING LGVideoAccess is pending

Finished in 32.4752 seconds

508 examples, 0 failures, 5 pending

You'll notice that the log prefixes such as Oct 13 20:10:03 My-MBP Specs[69607]: are gone, and I'm able to see the test copy.

I agree with your comment about disliking Test bundles - I do like my current route because I'm able to run this as part of a rake task which runs other RSpec and Jasmine tests. I will check out ioslib.

[mkanthan]

Since upgrading, I've been on Phonegap's ios-sim, and I believe I found an issue related to their logging which no longer logs to stdout: ios-control/ios-sim#181