Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async tests crash #341

Open
jaishankar opened this issue Jul 18, 2013 · 21 comments
Open

Async tests crash #341

jaishankar opened this issue Jul 18, 2013 · 21 comments
Labels

Comments

@jaishankar
Copy link

I am getting the crash very recently, in the last 10 days https://github.com/allending/Kiwi/issues/293

Exactly the same issue mentioned in 293

I am using kiwi 2.2

@jaishankar
Copy link
Author

I am trying this
[[expectFutureValue(theValue(successBlockCalled)) shouldEventually] beTrue];

and it crashes at KWProbePoller here [[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:delayInterval]];

It looks weird for me as it was working fine for me 10 to 15 days before.

@stepanhruda
Copy link
Member

Did you update Kiwi recently? Did you change the production code/spec recently? Can you paste the whole spec?

I tried

it(@"doesn't crash", ^{
    __block BOOL blockWasCalled = NO;

    dispatch_time_t delay = dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.3 * NSEC_PER_SEC));
    dispatch_after(delay, dispatch_get_main_queue(), ^(void){
        blockWasCalled = YES;
    });

    [[expectFutureValue(theValue(blockWasCalled)) shouldEventually] beYes];
});

and it works for me as expected.

@jaishankar
Copy link
Author

The following is the spec I m using

From the pod I am
Using Kiwi (2.2)
Using OHHTTPStubs (2.0.0)

I am getting the crash some time or other in the I have commented as Crash #

Very few times it is passing all. I dont think I have changed the code much, but I have updated the OHHTTPStubs
where the method name changed to OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request)
from the previous one.

describe(@"Verify the user stats service.", ^{

__block BOOL successBlockCalled = NO;
__block BOOL failureBlockCalled = NO;

beforeEach(^{
    successBlockCalled = NO;
    failureBlockCalled = NO;
});

afterEach(^{
    [OHHTTPStubs removeAllRequestHandlers];
});

//Success Response
it(@"Get user stats service should call success block when success response is obtained from the server.", ^{

    [OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request) {
        return YES;
    } withStubResponse:^OHHTTPStubsResponse *(NSURLRequest *request) {

        //Obtain the mock data
        NSData *data = [MyTestUtils cannedDataWithName:@"user_stats_success"];
        return [OHHTTPStubsResponse responseWithData:data statusCode:200 responseTime:0.5 headers:[MyTestUtils cannedHeaders]];
    }];

    [[MyService sharedInstance] getUserStatsWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
            successBlockCalled = YES;
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
            failureBlockCalled = YES;
    }];

   //Crash 1
    [[expectFutureValue(theValue(successBlockCalled)) shouldEventually] beTrue];

    //Crash 2
    [[expectFutureValue(theValue(failureBlockCalled)) shouldEventually] beFalse];
});

//Failure response
it(@"Get user stats service should call failure block when failure response is received from the server.", ^{

    [OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request) {
        return YES;
    } withStubResponse:^OHHTTPStubsResponse *(NSURLRequest *request) {

        //Obtain the mock data from the
        return [OHHTTPStubsResponse responseWithData:nil statusCode:404 responseTime:0.5 headers:[MyTestUtils cannedHeaders]];
    }];

    [[MyService sharedInstance] getUserStatsWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
            successBlockCalled = YES;
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
             failureBlockCalled = YES;
    }];

    //Crash 3 
    [[expectFutureValue(theValue(successBlockCalled)) shouldEventually] beFalse];
    //Crash 4
    [[expectFutureValue(theValue(failureBlockCalled)) shouldEventually] beTrue];
});    

});

@jaishankar
Copy link
Author

I think mine is related to the open issue here
https://github.com/allending/Kiwi/issues/300

@dyba
Copy link

dyba commented Aug 9, 2013

I don't see any Async tests in Kiwi. Can anyone point me to where I can find them?

@supermarin
Copy link

@dyba not sure i've got your question; are you looking for OCTests testing Kiwi async implementations,

or you're looking for a way to test your async code with Kiwi?
If latter, the helpers you want to use are expectFutureValue() together with shouldEventually

@dyba
Copy link

dyba commented Aug 9, 2013

@mneorr I'm looking for the tests for the Kiwi async implementations. I want to add a few tests of my own to understand the behavior of shouldEventually.

@dyba
Copy link

dyba commented Aug 9, 2013

"If no input sources or timers are attached to the run loop, this method exits immediately; otherwise, it runs the
receiver in the NSDefaultRunLoopMode by repeatedly invoking runMode:beforeDate: until the specified expiration
date." ( Italics mine )

Here is the KWProbePoller class as it is currently implemented:

// Kiwi/Classes/Core/KWProbePoller.m

- (BOOL)check:(id<KWProbe>)probe; {
    KWTimeout *timeout = [[KWTimeout alloc] initWithTimeout:self.timeoutInterval];

    while (self.shouldWait || ![probe isSatisfied]) {
        if ([timeout hasTimedOut]) {
            return [probe isSatisfied];
        }
        [[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:self.delayInterval]];
        [probe sample];
    }

    return YES;

So, the NSRunLoop will exit immediately because there are no sources or timers attached to it. By timer, I believe Apple meant an NSTimer and not a NSTimeInterval. Under the hood, KWProbePoller is using an NSTimeInterval.

My idea is to grab the run loop, set up a timer that polls every 0.1 second by default to check for the result of the expectation. After a given timeout, we return the result of the expectation.

- (BOOL)check:(id<KWProbe>)probe; {
    KWTimeout *timeout = [[KWTimeout alloc] initWithTimeout:self.timeoutInterval];

    NSRunLoop *currentRunLoop = [NSRunLoop currentRunLoop];
    NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:0.1
                                                      target:probe
                                                    selector:@selector(sample)
                                                    userInfo:nil
                                                     repeats:YES];
    [currentRunLoop addTimer:timer forMode:NSDefaultRunLoopMode];

    while (self.shouldWait || ![probe isSatisfied]) {
        if ([timeout hasTimedOut]) {
            [timer invalidate];
            return [probe isSatisfied];
        }
    }

    return YES;
}

This doesn't quite work yet and I'm trying to figure out why.

@dyba
Copy link

dyba commented Aug 9, 2013

Here's a first pass that works enough to substitute for the existing implementation. I'm still in the middle of testing it, but in case I don't get to finish, maybe someone can pick it up from here.

- (BOOL)check:(id<KWProbe>)probe {
    NSRunLoop *currentRunLoop = [NSRunLoop currentRunLoop];
    NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:0.1
                                                      target:probe
                                                    selector:@selector(sample)
                                                    userInfo:nil
                                                     repeats:YES];
    [currentRunLoop addTimer:timer
                     forMode:NSDefaultRunLoopMode];

    [currentRunLoop runUntilDate:[NSDate dateWithTimeIntervalSinceNow:self.timeoutInterval]];

    return [probe isSatisfied];
}    

@stepanhruda
Copy link
Member

Does this implementation abandon the runloop immediately if the probe is satisfied? I believe that is what the old one does and also something we want to bring down the test duration.

@dyba
Copy link

dyba commented Aug 11, 2013

@stepanhruda No, it does not. That's a good point. That makes me think we will need to build on the implementation I provided to add a CFRunLoopObserver to the run loop that checks when the probe has been satisfied and also reuse some of the existing implementation. When the probe has been satisfied, the run loop should exit. Here's another shot. I'm just thinking out loud. I can't verify if these tests run because I'm having Mach-O Linker Error problems that I'm trying to solve. Ugh, I'm not very savvy when it comes to these kind of compiled language errors.

- (BOOL)check:(id<KWProbe>)probe {
    __block BOOL shouldKeepRunning = YES;
    NSRunLoop *currentRunLoop = [NSRunLoop currentRunLoop];
    NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:0.1
                                                      target:probe
                                                    selector:@selector(sample)
                                                    userInfo:nil
                                                     repeats:YES];
    CFRunLoopObserverRef probeIsSatisfiedObserver =
    CFRunLoopObserverCreateWithHandler(kCFAllocatorDefault,
                                       kCFRunLoopAllActivities,
                                       true,
                                       0,
                                       ^(CFRunLoopObserverRef observer,
                                         CFRunLoopActivity activity) {
                                            if ([probe isSatisfied]) {
                                                shouldKeepRunning = NO;
                                            };
                                        });

    CFRunLoopAddObserver([currentRunLoop getCFRunLoop],
                         probeIsSatisfiedObserver,
                         kCFRunLoopCommonModes);
    [currentRunLoop addTimer:timer
                     forMode:NSDefaultRunLoopMode];

    while (shouldKeepRunning && [currentRunLoop runMode:NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:self.timeoutInterval]]) {
    }

    return [probe isSatisfied];
}

CFRunLoop Observer Reference, http://developer.apple.com/library/mac/documentation/CoreFoundation/Reference/CFRunLoopObserverRef/Reference/reference.html

@dyba
Copy link

dyba commented Aug 11, 2013

I'm getting Mach-O Linker Errors because I brought in the AFNetworking and OHHTTPStubs frameworks into the project to set up a test that mimics the real problem that got me started on this quest. I'll see if I can fix these errors first.

@dyba
Copy link

dyba commented Aug 11, 2013

LOL, that while loop has to be changed, otherwise it'll be an infinite loop if the probe is not satisfied.

@dyba
Copy link

dyba commented Aug 11, 2013

I got past the Mach-O linker errors and managed to get a passing spec for the problem I was working on. Here are just two of the specs that now pass. Of course, the last bit involves fixing the condition on the while loop to account for failing cases.

// Kiwi/Tests/Functiona/KWFunctionalAsyncTests.m

#import "Kiwi.h"
#import "OHHTTPStubs.h"
#import "OHHTTPStubsResponse.h"
#import "TimeTraveler.h"

SPEC_BEGIN(FunctionalAsyncTests)

describe(@"Async Tests", ^{
    it(@"passes", ^{
        TimeTraveler *timeTraveler = [[TimeTraveler alloc] initWithMessage:@"Taking off!" andURL:[NSURL URLWithString:@"http://example.com"]];
        NSString *newMessage = @"I have traveled in time!";

        [timeTraveler sendMessage:newMessage afterDelay:0.95];

        [[expectFutureValue(timeTraveler.message) shouldEventually] equal:newMessage];
    });

    it(@"passes for async tests", ^{
        TimeTraveler *timeTraveler = [[TimeTraveler alloc] initWithMessage:@"Taking off!" andURL:[NSURL URLWithString:@"http://example.com"]];

        [OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request) {
            return YES;
        } withStubResponse:^OHHTTPStubsResponse*(NSURLRequest *request) {
            OHHTTPStubsResponse *response = [OHHTTPStubsResponse responseWithFile:@"example.json"
                                                                      contentType:@"text/json"
                                                                     responseTime:0.95];
            return response;
        }];

        [timeTraveler fetchAsynchronousUsingMessage:@"I did it!"];

        [[expectFutureValue(timeTraveler.message) shouldEventually] equal:@"I did it!"];
    });
});

And the test class I used:

//
//  TimeTraveler.h
//  Kiwi
//
//  Created by Daniel Dyba on 8/8/13.
//  Copyright (c) 2013 Allen Ding. All rights reserved.
//

#import <Foundation/Foundation.h>
#import "AFHTTPClient.h"

@interface TimeTraveler : AFHTTPClient

@property (nonatomic, retain) NSString *message;

- (id)initWithMessage:(NSString *)aMessage andURL:(NSURL *)aURL;
- (void)sendMessage:(NSString *)aMessage
         afterDelay:(NSTimeInterval)aTimeInterval;
- (void)fetchAsynchronousUsingMessage:(NSString *)aMessage;

@end

//
//  TimeTraveler.m
//  Kiwi
//
//  Created by Daniel Dyba on 8/8/13.
//  Copyright (c) 2013 Allen Ding. All rights reserved.
//

#import "TimeTraveler.h"

@implementation TimeTraveler

- (id)initWithMessage:(NSString *)aMessage andURL:(NSURL *)aURL {
    self = [super initWithBaseURL:aURL];

    if (self)
        _message = aMessage;

    return self;
}

- (void)fetchAsynchronousUsingMessage:(NSString *)aMessage {
    [self getPath:@"example.json"
       parameters:nil
          success: ^(AFHTTPRequestOperation *operation, id responseObject) {
              self.message = aMessage;
              NSLog(@"Success!");
          }
          failure: ^(AFHTTPRequestOperation *operation, NSError *error) {
              NSLog(@"Failure!");
          }];
}

- (void)sendMessage:(NSString *)aMessage
         afterDelay:(NSTimeInterval)aTimerInterval {
    [self performSelector:@selector(setMessage:)
               withObject:aMessage
               afterDelay:aTimerInterval];
}

@end

@dyba
Copy link

dyba commented Aug 12, 2013

Here is my progress so far. Check out the AddAsyncTests branch on my fork. I could really use some help with the implementation.

@tomzilla
Copy link

tomzilla commented Oct 9, 2013

What's the status on this? Getting this crash on 2.2.2.

@dyba
Copy link

dyba commented Oct 9, 2013

@tomzilla I haven't had a chance to continue my progress because I've been busy with other things. Feel free to take a stab at solving the problem. I pushed code for others to look at as I mentioned in my last comment.

@bilby91
Copy link

bilby91 commented Dec 18, 2013

Im having the same issue. I reproduce it with versions 2.2.2 and 2.2.3.

[[expectFutureValue(theValue(requestOperation.HTTPRequestOperation.response.statusCode)) shouldEventually] equal:@200];

Crashes here:
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:self.delayInterval]];

@yas375
Copy link
Contributor

yas375 commented Dec 26, 2013

I've tried to bring OHHTTPStubs to the project I'm working on and was getting the same error. We have a special class for wrapping up network requests. Tt also parses json and stores the data to CoreData. If I'm not wrong was getting this crash when on another thread we were executing NSManagedObjectContext#performBlockAndWait:...

Sorry, I can't be much more explicit because after spending a few hours on it I have rejected the idea of such testing and have added tests in another way. So can't recheck stack traces. If somebody have time to fix it, I can try to reproduce it on a test project...
Thanks.

@bilby91
Copy link

bilby91 commented Dec 26, 2013

What path did you finaly take for testing? BTW, are you using an open source lib for json parsing and storing in core data?

@yas375
Copy link
Contributor

yas375 commented Dec 26, 2013

@bilby91 I know what method is invoked from completion block when network request is completed. So I just invoke this method in my tests. It's not very good that tests depend on internal implementation, so I hope someday I'll switch to OHHTTPStubs again :)

p.s. we are using NSJSONSerialization and some custom code for mapping (using KVC and runtime). You could also take a look at Mantle, DVJSONMapping (we are using it in another project - it works :)), ObjectMapper (just googled it, haven't used)...

@supermarin supermarin added the Bug label Mar 1, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants