Merge pull request msavin#37 from msavin/v3

V3

CHANGES.md

@@ -0,0 +1,115 @@
+<img align="right" width="220" src="https://github.com/msavin/stevejobs/blob/master/ICON.png?raw=true" />
+
+# Steve Jobs: What's New in 3.0
+
+## The Simple Jobs Queue That Just Works
+
+Meteor evolved application development by going from past-time by default to real-time by default. The Steve Jobs package takes the next leap forward by letting your application run in future-time. 😁
+
+The Steve Jobs package allows you to schedule tasks at a future date, in a way that is friendly to the Meteor framework. It provides a wide set of tools to help you get creative, while automating everything behind it.
+
+## Repeating Jobs (!!!)
+
+The most requested feature was also the trickiest to implement, due to how the rest of the queue was built. I didn't want to hack in a completely new set of features on top of what already exists. Fortunately, it turns out there's a really simple way to achieve this effect while staying aligned with how the rest of the queue works.
+
+```javascript
+Jobs.register({
+    "syncData": function () {
+        var self = this;
+        var data = getDataFromSomewhere();
+
+        if (data) {
+            var doc = MyCollection.insert(data);
+
+            if (doc) {
+                self.replicate({
+                    in: {
+                        minutes: 5
+                    }
+                })
+
+                self.success(doc)
+            } else {
+                self.reschedule({
+                    in: {
+                        seconds: 30
+                    }
+                })
+            }
+        } else {
+            self.reschedule({
+                in: {
+                    minutes: 5
+                }
+            })
+        }
+    }
+})
+```
+
+The job above will try to get data through some means, and then:
+  - If it receives the data as expected, it will try to insert it into a MongoDB collection
+    - If the insert is successful, it will `replicate` the job and tell it to run in 5 minutes, and then mark the original as a `success`
+    - If the insert is not successful, the job will `reschedule` itself to run again in 30 seconds
+  - If it does not receive the data as expected, it will `reschedule` itself to run again in 5 minutes
+
+`this.replicate` basically replicates the job and its arguments, while allowing you to set a new configuration for it. This enables you to repeat a job as many times as you wish, while dynamically setting the conditions for how it should happen.
+
+It's important to use `this.replicate` instead of `this.reschedule` to repeat a job because each job keeps track of its history. If you reschedule a job too many times, it's document would become humongous, which could have consequences. Additionally, using `this.replicate` makes it easier for your clear resolved job documents in the future.
+
+For jobs that that run very frequently, you can also use the new `this.remove` feature to remove the document from the database rather than just mark it as complete.
+
+## More Tools for Designing Your Jobs
+
+When you register a job with `Jobs.register`, you can access a wide array of tools to make sure the job runs exactly the way you want it to:
+ - `this.document` - access the cached document of the current job
+ - `this.set` & `this.get` - persistent state management for the current job
+ - `this.success` - mark the job as successful
+ - `this.failure` - mark the job as having failed
+ - `this.reschedule` - tell the job to run again later
+ - `this.replicate` - replicate the job with the same arguments
+ - `this.remove` - remove the job from the collection
+
+Every job must be resolved with `this.success`, `this.failure`, `this.reschedule`, or `this.remove`, otherwise the queue will log an error and stop running. This is to ensure that a job does not end up looping infinitely.
+
+Each job can hold its own state thanks ot `this.set` and `this.get` - meaning if you experience an interruption, you can get the job to pick up where it left off. It can also be used to display things like progress bars.
+
+Additionally, `this.failure` is automatically called inside of a try `try catch` block when the code has an error. If your code works fine and the job "failed" for reasons other than code execution, I suggest using `this.reschedule` instead. 
+
+## New Configuration Options
+
+`Jobs.configure` now allows you to customize three more core functions of the Jobs package: `setServerId`, `getDate` and `log`.
+
+With `getServerId`, you can now specify the mechanism for generating a unique server ID. By default, this uses `Random.id()`, but after @gary-menzel's suggestion, it made sense to open this function for customization so that it can be integrated with the server ID that you hosting service may assign.
+
+With `getDate`, you can now specify how a new Date object should be initialized. By default, the function will return `new Date()`. With this option, you can, for example, return a Date object that has a future date, to create a "time travel" effect. Thanks to the person who suggested this.
+
+With `log`, you can configure how your application should log items. By default, the function will use `console.log`. 
+
+## Smarter MongoDB Querying
+
+Of all the pleasures that MongoDB offers, peace of mind is not one of them. 
+
+First, it can take a bit of time for the writes to be reflected in the reads, and that could make jobs run twice. 
+
+This was resolved by adding an extra condition to the MongoDB queries: the document must meet this criteria, _and_ its `_id` must not be that of the job that had just run.
+
+Second, it turns out that MongoDB's upsert function may not be so reliable - if you run a few upserts at once, MongoDB might just insert all the documents. This is probably related to the first issue.  This created a problem with the `dominator` function, as the queue might get confused as to which server is active.
+
+This has been resolved by making the `serverId` field unique. It looks like MongoDB becomes more diligent when you set a unique field.
+
+## What's Next?
+
+As is - the Steve Jobs package does its job well, and works great with Meteor.
+
+I'm excited about transactions coming MongoDB 4.0. Along with Write Concerns, Retryable Writes, and the new storage engine, this can be used to make the queue _really_ reliable.
+    - In some jobs, you might need to run two database operations, such  `this.replicate` and `this.success`, to successfully resolve the job. It would be nice if the two can be combined to assure that both actions happened successfully.
+    - It could also be helpful in designing a mechanism to keep track of many jobs running across many servers.
+
+MongoDB 4.0 is coming in the summer, so I will evaluate then whether to keep evolving the project or to simply maintain what it does now.
+
+The idea is, this could grow into a reliable queue that can run many jobs at once and scale horizontally. It would not be the fastest solution, but it may be so reliable, scalable and developer friendly, that speed would seem overrated.
+
+If that were achieved, the next step would be to build an interface and REST API to let anyone run this as a standalone service.
+
+**With that said, your help can go a long way!** I'm looking for someone to design and implement the testing strategy, and for help in taking on the challenge of scaling the queue to run many jobs at once. If you can help with this, or if you have a different angle, do reach out!

LICENSE.md

@@ -1,4 +1,4 @@
-Copyright 2017 Max Savin
+© 2018 Max Savin
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

README.md

@@ -12,7 +12,7 @@ Run scheduled tasks with Steve Jobs, the simple jobs queue made just for Meteor.
  - Failed jobs are retried on server restart
  - No third party dependencies
 
-**The upcoming 3.0 will come with repeating jobs and more.** In the meantime, the current version can help you run hundreds of jobs in seconds with minimal CPU impact, making it a reasonable choice for many applications. To get started, check out the Quick Start below, take a look at the <a href="https://github.com/msavin/SteveJobs/wiki">**documentation**</a>, and/or try the <a href="http://jobsqueue.herokuapp.com">**live demo**</a>.
+**The new 3.0 features repeating jobs and more improvements.** It can run hundreds of jobs in seconds with minimal CPU impact, making it a reasonable choice for many applications. To get started, check out the Quick Start below, take a look at the <a href="https://github.com/msavin/SteveJobs/wiki">**documentation**</a>, and/or try the <a href="http://jobsqueue.herokuapp.com">**live demo**</a>.
 
 ## Developer Friendly UI and API
 

package/package.js

@@ -1,7 +1,7 @@
 Package.describe({
 	name:    "msavin:sjobs",
 	summary: "The simple jobs queue that just works [synced, schedule, tasks, background, later, worker, cron]",
-	version: "2.0.1",
+	version: "3.0.2",
 	documentation: "README.md",
 	git:     "https://github.com/msavin/SteveJobs.git",
 });

package/server/README.md

@@ -0,0 +1,16 @@
+# Steve Jobs / Server / Folder Contents
+
+The `api.js` file imports all the package functionality and exposes a public API for developers to use on the server. 
+
+Features of the public API: 
+ - every function has argument checking for maximum security and reliability
+ - every function is private and can only be called on the server
+ - `JobsInternal` exposes package internals for debugging
+
+-----
+
+The public API obtains its functionality from the `imports` folder, which is broken up into three folders:
+ - `actions` - all the actions possible with-in the queue, such as adding a job, executing a job, rescheduling a job, etc
+ - `operator` - automatically checks for whichs jobs to run, ensures the right conditions for them to run in, and runs them
+ - `utilities` - helper functions and shared functionality for `actions` and `operator`
+ - `startup` - startup functions for the server, used to start Operator

package/server/api.js

@@ -1,8 +1,9 @@
 import { Actions } from './imports/actions'
 import { Utilities } from './imports/utilities'
 import { Operator } from './imports/operator'
+import './imports/startup'
 
-Jobs = {}
+var Jobs = {}
 
 // Configure the package (optional)
 
@@ -11,7 +12,10 @@ Jobs.configure = function (config) {
 		autoStart: Match.Maybe(Boolean),
 		interval: Match.Maybe(Number),
 		startupDelay: Match.Maybe(Number),
-		maxWait: Match.Maybe(Number)
+		maxWait: Match.Maybe(Number),
+		setServerId: Match.Maybe(Function),
+		getDate: Match.Maybe(Function),
+		log: Match.Maybe(Function)
 	})
 
 	Object.keys(config).forEach(function (key) {
@@ -126,30 +130,57 @@ Jobs.execute = function (jobId, callback, force) {
 
 Jobs.reschedule = function (jobId, config) {
 	check(jobId, String)
-	check(config, {
+	if (config) check(config, {
 		date: Match.Maybe(Date),
 		in: Match.Maybe(Object),
 		on: Match.Maybe(Object),
 		priority: Match.Maybe(Number),
+		callback: Match.Maybe(Function)
 	})
 
 	return Actions.reschedule(jobId, config);
 }
 
+// Replicate a job to run it again later
+
+Jobs.replicate = function (jobId, config) {
+	check(jobId, String)
+	if (config) check(config, {
+		date: Match.Maybe(Date),
+		in: Match.Maybe(Object),
+		on: Match.Maybe(Object),
+		data: Match.Maybe(Object),
+		priority: Match.Maybe(Number),
+	})
+
+	return Actions.replicate(jobId, config);
+}
+
 // Clear resolved jobs - or all of them 
 
-Jobs.clear = function (state, name) {
+Jobs.clear = function (state, name, callback) {
 	check(state, Match.OneOf(undefined, String, [String]))
 	check(name, Match.Optional(String))
+	check(callback, Match.Optional(Function))
+
+	return Actions.clear(state, name, callback);
+}
 
-	return Actions.clear(state, name);
+// Remove the job
+
+Jobs.remove = function (jobId, callback) {
+	check(jobId, String)
+	check(callback, Match.Maybe(Function))
+
+	return Actions.remove(jobId, callback);
 }
 
 // Internals for debugging
 
-var JobsInternal = {}
-JobsInternal.Actions = Actions;
-JobsInternal.Utilities = Utilities;
-JobsInternal.Operator = Operator;
+var JobsInternal = {
+	Actions: Actions,
+	Utilities: Utilities,
+	Operator: Operator
+}
 
 export { Jobs, JobsInternal }

package/server/imports/actions/add/index.js

@@ -1,27 +1,34 @@
-import { Utilities } from '../../utilities'
-import { generateDoc } from './generateDoc.js'
-import { processInput } from './processInput.js'
+import { Utilities } from "../../utilities"
+import { processArguments } from "./processArguments.js"
 
 var add = function () {
-	// 1. Convert arguments to array + prepare necessary data
-	var userEntry = processInput(arguments);
+	// 0. Prepare variables
+	var error, result;
+
+	// 1. Process arguments + prepare the data
+	var input = processArguments(arguments);
 
 	// 2. Generate job document
-	var jobInfo = generateDoc(userEntry);
+	var jobDoc = Utilities.helpers.generateJobDoc(input);
 
-	// 3. Add to the database
-	var jobId = Utilities.collection.insert(jobInfo);
+	// 3. Insert the job document into the database
+	var jobId = Utilities.collection.insert(jobDoc);
 
-	// 4. Simulate the document (this saves us a database request)
-	var simulatedDoc = jobInfo;
-	simulatedDoc._id = jobId;
+	// 4. Simulate the document (this might save us a database request in some places)	
+	if (jobId) {
+		result = jobDoc;
+		result._id = jobId;
+		result._simulant = true;
+	} else {
+		error = true;
+	}
 
 	// 5. Mission accomplished
-	if (typeof userEntry.config.callback === "function") {
-		return userEntry.config.callback(undefined, simulatedDoc);
-	} else {
-		return simulatedDoc;
+	if (typeof input.config.callback === "function") {
+		input.config.callback(error, result);
 	}
+
+	return result;
 }
 
 export { add }

package/server/imports/actions/add/processArguments.js

@@ -0,0 +1,42 @@
+var checkConfig = function (input) {
+	var result = false,
+		lastArgument = input[input.length - 1],
+		keywords = ["in", "on", "priority", "date", "data", "callback"];
+
+	if (typeof lastArgument === "object") {
+		keywords.forEach(function (keyword) {
+			if (lastArgument[keyword]) {
+				result = true;
+			}
+		});
+	}
+
+	return result;
+}
+
+var processArguments = function (args) {
+	var output = {},
+		args = Array.prototype.slice.call(args);
+
+	output.name = function () {
+		var name = args.shift();
+		return name;
+	}()
+
+	output.config = function () {
+		if (checkConfig(args)) {
+			var config = args.pop();
+			return config;
+		} else {
+			return {};
+		}
+	}()
+
+	output.arguments = function () {
+		return args;
+	}()
+
+	return output;
+}
+
+export { processArguments }