Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/marker pagination fixes #298

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Commits on Sep 12, 2016

  1. Configuration menu
    Copy the full SHA
    1adc4ce View commit details
    Browse the repository at this point in the history
  2. test: Assert response code in FeedTagTest

    This makes it easier to debug issues that case 500 errors.
    reftel committed Sep 12, 2016
    Configuration menu
    Copy the full SHA
    24c3606 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2016

  1. test: align AtomHopperJettyServerBuilder impls

    The implementation of AtomHopperJettyServerBuilder in test-suite did not
    have a constructor that takes config file location as argument, as the
    one in ah-jetty-server did. This change syncs the one in test-suite up
    with the one in ah-jetty-server.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    c03a5d7 View commit details
    Browse the repository at this point in the history
  2. test: Programs for verifying following by marker

    MarkerRaceConditionTest posts multiple entries in parallel, while
    following the feed using the marker mechanism. The code is written to
    try to maximize the risk of having multiple entries being handled in
    parallel by the server. If any entry is written with an older timestamp
    than the latest, then that entry will be missed, and this will be
    revealed by the program.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    d072422 View commit details
    Browse the repository at this point in the history
  3. hibernate: Use dateLastUpdated for pagination

    When using a marker for pagination, entries were selected by comparing
    their dateLastUpdated with the creation date of the marker, which causes
    wrong results when using an updated marker.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    db207d9 View commit details
    Browse the repository at this point in the history
  4. Use Java 8

    This is required for high-resolution time stamps in Hibernate.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    b0a93f9 View commit details
    Browse the repository at this point in the history
  5. Use Hibernate 5.1

    This, together with the hibernate-java8 module, enables use of
    high-resolution timestamps.
    
    In 5.2, the criteria support has been deprecated, and warnings are
    logged on each use of them. Migrating to a different mechanism is a
    large, separate task, so stay on 5.1 for now.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    68552a6 View commit details
    Browse the repository at this point in the history
  6. Use higher precision time stamps internally

    This reduces the probability of entries sharing the same timestamps.
    Shared timestamps cause issues with pagination.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    eac408d View commit details
    Browse the repository at this point in the history
  7. hibernate: Use database timestamps for entries

    If application servers have clocks that are not in sync, then it is
    possible for timestamps to not be increasing. If there are two
    application servers with out-of-sync clocks, where the clock of server A
    is a bit ahead of the clock of server B, then it's possible that an
    entry written by A gets a higher timestamp than en entry written at a
    later point by server B. If a consumer read the entry written by server
    A, and then used it as a marker, then it would not receive the entry
    written by server B.
    
    This change makes all application servers ask the database server for
    the clock, similar to how it is done in the Postgres adapter.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    59d016c View commit details
    Browse the repository at this point in the history
  8. hibernate: Lock the feed when adding entries

    Otherwise, there's a race condition where newer entries can get older
    timestamps: if server A starts a transaction, gets a timestamp, but then
    gets blocked, and server B then starts a transaction, gets a higher
    timstamp, and commits, then when server A resumes, its entry will be
    written at a later time then the entry from server B, but with a lower
    timestamp. A consumer that read the entry from server B and used it as a
    marker would then not get the entry from server A. Locking the feed
    before getting the timestamp and holding it until the commit solves this
    issue, at the cost of serializing all writes to a feed.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    3091020 View commit details
    Browse the repository at this point in the history
  9. hibernate: Mark timesetamps unique

    Pagination doesn't work well when a marker has the same timestamp as
    other entries. If one producer writes an entry, a consumer reads it, and
    then another producer writes a new entry, and this gets the same
    timestamp as the first entry, then it's not clear whether the consumer
    should get this second entry or not when using the first entry as
    marker. If entries with the same timestamp as the marker should be
    included, then consumers may get entries multiple times if they share
    timestamp with the marker. If entries with the same timestamp as the
    marker should not be included, then consumers may miss entries.
    
    Using secondary sorting on ID also doesn't work, since it then is
    possible for a new entry that happen to get a lower-sorting ID to be
    considered older than an existing entry with a higher-sorting ID.
    
    This commit prevents these error conditions by forbidding sharing of
    timestamps between entries in the same feed. That way, a producer that
    happens to get the same timestamp as an earlier entry will have to
    retry, giving it a new timestamp. This ensures that consumers get each
    entry exactly once.
    
    Due to the high resolution time stamps and the lock on the feed,
    collisions should in practice never happen.
    reftel committed Oct 5, 2016
    Configuration menu
    Copy the full SHA
    77393d0 View commit details
    Browse the repository at this point in the history