Browse Source

pipeline: Directed graph execution model

Our current pipelinelib based jobs require repos to conform to a number
of rigid conventions: assume the repo contains source for only a single
application, build "test" variant, run "test" variant, build
"production" variant, helm deploy/test, publish under a single tag name.
These jobs also assume all of these operations need to be performed
linearly.

While this design was sufficient for our very first use cases, its
convention based design it already proving prohibitively inflexible. For
example, teams maintaining repos that contain multiple interrelated
applications cannot build and test these applications as independent
images; Teams wanting to execute multiple test suites would have to wrap
them in a single entrypoint and implement their own concurrency should
they need it; Etc.

Instead of Release Engineering maintaining a new specialized pipeline
job for each team that performs only slightly different permutations of
the same operations (resulting in duplication of job definitions and a
large maintenance burden), we can instead establish a configuration
format and interface by which teams provide their own pipeline
compositions.

This initial commit in a series of pipeline related commits implements
two fundamental components to support a CI/CD pipeline that can execute
any number of user-defined variant build/test/publish/deploy stages and
steps in a safely concurrent model: a directed-graph based execution
model, and name bindings for stage outputs. The former provides the
model for composing stage execution, and the latter provides a decoupled
system for defining what outputs each subsequent stage operates upon.

First, an `ExecutionGraph` class that can represent a directed acyclic
graph given a number of linearly defined arcs (aka branches/edges). This
component will allow users to provide the overall execution flow as
separate linear processes but allow parallel branches of the execution
graph to be scheduled concurrently.

Example:

    /* To represent a graph with separate parallel branches like:
     *
     *   a       x
     *     ⇘   ⇙
     *       b
     *     ⇙   ⇘
     *   y       c
     *     ⇘   ⇙
     *       z
     *
     * One only needs to provides each linear execution arc
     */
    def graph = new ExecutionGraph([["a", "b", "c", "z"], ["x", "b", "y", "z"]])

    /* The ExecutionGraph can solve how those arcs intersect and how the
     * nodes can be scheduled with a degree of concurrency that Jenkins
     * allows.
     */
    graph.stack() // => [["a", "x"], ["b"], ["y", "c"], ["z"]]

Second, a set of context classes for managing immutable global and local
name/value bindings between nodes in the graph. Effectively this will
provide a way for pipeline stages to safely and deterministically
consume inputs from previous stages along the same branch, and to
provide their own outputs for subsequent stages to consume.

For example, one stage called "build" that builds a container image will
save the image ID in a predetermined local binding called `.imageID` and
a subsequent "publish" stage configured by the user can reference that
image by `${build.imageID}`.

Once a value is bound to a name, that name cannot be reused; bindings
are immutable. Node contexts are only allowed to access namespaces for
nodes that precede them in same branch of the graph, ensuring
deterministic behavior during parallel graph branch execution. See unit
tests for `ExecutionContext` for details on expected behavior.

Put together, these two data structures can constitute an execution
"stack" of sorts that can be safely mapped to Jenkins Pipeline stages,
and make use of parallel execution for graph branches. Specifically, the
`ExecutionGraph.stack()` method is implemented to yield each set of
independent stack "frames" in topological sort order which can safely be
scheduled to run in parallel.

Bug: T210267
Change-Id: Ic5d01bf54c703eaf14434a36f1e2b3e276b48b6f
Dan Duvall 5 months ago
parent
commit
858c26317b

+ 25
- 1
build.gradle View File

@@ -1,3 +1,6 @@
1
+import org.gradle.api.tasks.testing.logging.TestLogEvent
2
+import org.gradle.api.tasks.testing.logging.TestExceptionFormat
3
+
1 4
 apply plugin: 'groovy'
2 5
 
3 6
 repositories {
@@ -6,6 +9,7 @@ repositories {
6 9
 
7 10
 dependencies {
8 11
   compile 'org.codehaus.groovy:groovy-all:2.4.11'
12
+  compile 'com.cloudbees:groovy-cps:1.24'
9 13
   testCompile 'junit:junit:4.12'
10 14
 }
11 15
 
@@ -29,7 +33,27 @@ groovydoc {
29 33
 
30 34
 test {
31 35
   testLogging {
32
-    exceptionFormat = 'full'
36
+    exceptionFormat TestExceptionFormat.FULL
37
+
38
+    events TestLogEvent.PASSED,
39
+           TestLogEvent.SKIPPED,
40
+           TestLogEvent.FAILED
41
+
42
+    info {
43
+      events TestLogEvent.STARTED,
44
+             TestLogEvent.PASSED,
45
+             TestLogEvent.SKIPPED,
46
+             TestLogEvent.FAILED,
47
+             TestLogEvent.STANDARD_OUT,
48
+             TestLogEvent.STANDARD_ERROR
49
+    }
50
+
51
+    debug.events = info.events
52
+  }
53
+
54
+  // Can be used for remote debugging in IntelliJ
55
+  if (System.getProperty('DEBUG', '0') == '1') {
56
+    jvmArgs '-Xdebug', '-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=9099'
33 57
   }
34 58
 }
35 59
 

+ 232
- 0
src/org/wikimedia/integration/ExecutionContext.groovy View File

@@ -0,0 +1,232 @@
1
+package org.wikimedia.integration
2
+
3
+import com.cloudbees.groovy.cps.NonCPS
4
+import org.codehaus.groovy.GroovyException
5
+
6
+import org.wikimedia.integration.ExecutionGraph
7
+
8
+/**
9
+ * Provides execution context and value bindings to graph nodes. Each node's
10
+ * context (see {@link ofNode()}) provides methods for saving its own values
11
+ * and bindings for accessing values saved by ancestor nodes.
12
+ *
13
+ * These contexts can be used during graph stack evaluation to safely pass
14
+ * values from ancestor nodes to descendent nodes, and to keep nodes from
15
+ * different branches of execution to access each other's values.
16
+ */
17
+class ExecutionContext implements Serializable {
18
+  private Map globals = [:]
19
+  private ExecutionGraph graph
20
+
21
+  ExecutionContext(executionGraph) {
22
+    graph = executionGraph
23
+  }
24
+
25
+  /**
26
+   * Create and return an execution context for the given node.
27
+   */
28
+  NodeContext ofNode(node) {
29
+    new NodeContext(node, graph.ancestorsOf(node))
30
+  }
31
+
32
+  /**
33
+   * Returns the names of all values bound by node contexts.
34
+   */
35
+  List getAllKeys() {
36
+    def keys = []
37
+
38
+    for (def ns in globals) {
39
+      for (def key in globals[ns]) {
40
+        keys.add("${ns}.${key}")
41
+      }
42
+    }
43
+
44
+    keys
45
+  }
46
+
47
+  /**
48
+   * Provides an execution context for a single given node that can resolve
49
+   * bindings for values stored by ancestor nodes, set its own values, and
50
+   * safely interpolate user-provided strings.
51
+   *
52
+   * @example
53
+   * Given a graph:
54
+   * <pre><code>
55
+   *   def context = new ExecutionContext(new ExecutionGraph([
56
+   *     ["a", "b", "c", "d", "e", "f"],
57
+   *     ["x", "d", "y", "f"],
58
+   *   ]))
59
+   * </code></pre>
60
+   *
61
+   * @example
62
+   * Values can be bound to any existing node.
63
+   * <pre><code>
64
+   *   context.ofNode("a")["foo"] = "afoo"
65
+   *   context.ofNode("a")["bar"] = "abar"
66
+   *   context.ofNode("b")["foo"] = "bfoo"
67
+   *   context.ofNode("x")["bar"] = "xbar"
68
+   * </code></pre>
69
+   *
70
+   * @example
71
+   * Those same values can be accessed in contexts of descendent nodes, but
72
+   * not in contexts of unrelated nodes.
73
+   * <pre><code>
74
+   *   assert context.ofNode("c").binding("a", "foo") == "afoo"
75
+   *   assert context.ofNode("c").binding("a", "bar") == "abar"
76
+   *   assert context.ofNode("c").binding("b", "foo") == "bfoo"
77
+   *
78
+   *   assert context.ofNode("c").binding("x", "bar") == null
79
+   *
80
+   *   assert context.ofNode("e").binding("a", "foo") == "afoo"
81
+   *   assert context.ofNode("e").binding("a", "bar") == "abar"
82
+   *   assert context.ofNode("e").binding("b", "foo") == "bfoo"
83
+   *   assert context.ofNode("e").binding("x", "bar") == "xbar"
84
+   * </code></pre>
85
+   *
86
+   * @example
87
+   * Leveraging all of the above, user-provided configuration can be safely
88
+   * interpolated.
89
+   * <pre><code>
90
+   *   assert (context.ofNode("c") % 'w-t-${a.foo} ${b.foo}') == "w-t-afoo bfoo"
91
+   *   assert (context.ofNode("c") % 'w-t-${a.bar}') == "w-t-abar"
92
+   *   assert (context.ofNode("x") % 'w-t-${x.bar}') == 'w-t-${x.bar}'
93
+   * </code></pre>
94
+   */
95
+  class NodeContext implements Serializable {
96
+    final VAR_EXPRESSION = /\$\{(\w*\.\w+)\}/
97
+
98
+    final def node
99
+    final Set ancestors
100
+
101
+    NodeContext(contextNode, contextAncestors) {
102
+      globals[contextNode] = globals[contextNode] ?: [:]
103
+
104
+      node = contextNode
105
+      ancestors = contextAncestors
106
+    }
107
+
108
+    /**
109
+     * Binds a value to a name in the globals store under the node's
110
+     * namespace. The value may later be retrieved by any descendent node's
111
+     * context using {@link binding()}.
112
+     */
113
+    @NonCPS
114
+    void bind(String key, value)
115
+      throws NameAlreadyBoundException {
116
+
117
+      if (globals[node].containsKey(key)) {
118
+        throw new NameAlreadyBoundException(key: key)
119
+      }
120
+
121
+      globals[node][key] = value
122
+    }
123
+
124
+    /**
125
+     * Retrieves a value previously bound using {@link bind()} to this node's
126
+     * context. If the given key is not found under this node's namespace, a
127
+     * {@link NameNotFoundException} is thrown.
128
+     */
129
+    def binding(String key)
130
+      throws NameNotFoundException {
131
+
132
+      if (!globals[node].containsKey(key)) {
133
+        throw new NameNotFoundException(ns: node, key: key)
134
+      }
135
+
136
+      globals[node][key]
137
+    }
138
+
139
+    /**
140
+     * Retrieves a value previously bound using {@link bind()} under the given
141
+     * ancestor node's namespace and name.
142
+     */
143
+    def binding(ancestorNode, String key)
144
+      throws NameNotFoundException, AncestorNotFoundException {
145
+
146
+      if (!(ancestorNode in ancestors)) {
147
+        throw new AncestorNotFoundException(ancestor: ancestorNode, node: node)
148
+      }
149
+
150
+      if (!globals[ancestorNode].containsKey(key)) {
151
+        throw new NameNotFoundException(ns: ancestorNode, key: key)
152
+      }
153
+
154
+      globals[ancestorNode][key]
155
+    }
156
+
157
+    /**
158
+     * Returns all objects bound to the given name under any node namespace.
159
+     * This should only be used at the end of an execution graph.
160
+     */
161
+    List getAll(String key) {
162
+      globals.findAll { it.value[key] != null }.collect { it.value[key] }
163
+    }
164
+
165
+    /**
166
+     * Operator alias for {@link binding(String)} or, if a "namespace.key" is
167
+     * given, {@link binding(def, String)}.
168
+     */
169
+    def getAt(String key) {
170
+      def keys = key.split(/\./)
171
+
172
+      if (keys.size() > 1) {
173
+        if (keys[0] == "") {
174
+          return binding(keys[1])
175
+        }
176
+
177
+        return binding(keys[0], keys[1])
178
+      }
179
+
180
+      return binding(key)
181
+    }
182
+
183
+    /**
184
+     * Interpolates the given string by substituting all symbol expressions
185
+     * with values previously bound by ancestor nodes.
186
+     */
187
+    @NonCPS
188
+    String interpolate(String str) {
189
+      str.replaceAll(VAR_EXPRESSION) { _, key ->
190
+        this[key]
191
+      }
192
+    }
193
+
194
+    /**
195
+     * Operator alias for {@link interpolate()}.
196
+     */
197
+    String mod(String str) {
198
+      interpolate(str)
199
+    }
200
+
201
+    /**
202
+     * Operator alias for {@link bind()}.
203
+     */
204
+    void putAt(String key, value) {
205
+      bind(key, value)
206
+    }
207
+  }
208
+
209
+  class AncestorNotFoundException extends GroovyException {
210
+    def ancestor, node
211
+
212
+    String getMessage() {
213
+      "cannot access '${ancestor}.*' values since '${node}' does not follow it in the graph '${graph}'"
214
+    }
215
+  }
216
+
217
+  class NameNotFoundException extends GroovyException {
218
+    def ns, key
219
+
220
+    String getMessage() {
221
+      "no value bound for '${ns}.${key}'; all bound names are: ${getAllKeys().join(", ")}"
222
+    }
223
+  }
224
+
225
+  class NameAlreadyBoundException extends GroovyException {
226
+    def key
227
+
228
+    String getMessage() {
229
+      "'${node}' already has a value assigned to '${key}'"
230
+    }
231
+  }
232
+}

+ 342
- 0
src/org/wikimedia/integration/ExecutionGraph.groovy View File

@@ -0,0 +1,342 @@
1
+package org.wikimedia.integration
2
+
3
+import com.cloudbees.groovy.cps.NonCPS
4
+
5
+/**
6
+ * Represents a directed acyclic graph (DAG) for defining pipeline stage
7
+ * dependencies and scheduling them in a parallel topological-sort order.
8
+ *
9
+ * An {@link ExecutionGraph} is constructed by passing sets of arcs (aka
10
+ * edges/branches) that may or may not intersect on common nodes.
11
+ *
12
+ * @example
13
+ * A graph such as:
14
+ * <pre><code>
15
+ *   a       w       z
16
+ *     ⇘   ⇙
17
+ *       b       x
18
+ *     ⇙   ⇘   ⇙
19
+ *   c       y
20
+ *     ⇘   ⇙
21
+ *       d
22
+ * </code></pre>
23
+ *
24
+ * Can be constructed any number of ways as long as all the arcs are
25
+ * represented in the given sets.
26
+ *
27
+ * <pre><code>
28
+ * def graph = ExecutionGraph([
29
+ *   ["a", "b", "c", "d"], // defines edges a → b, b → c, c → d
30
+ *   ["w", "b", "y"],      // defines edges w → b, b → y
31
+ *   ["x", "y", "d"],      // defines edges x → y, y → d
32
+ *   ["z"],                // defines no edge but an isolate node
33
+ * ])
34
+ * </code></pre>
35
+ *
36
+ * @example
37
+ * The same graph could be constructed this way.
38
+ *
39
+ * <pre><code>
40
+ * def graph = ExecutionGraph([
41
+ *   ["a", "b", "y"],
42
+ *   ["w", "b", "c", "d"],
43
+ *   ["x", "y"],
44
+ *   ["z"],
45
+ * ])
46
+ * </code></pre>
47
+ *
48
+ * @example
49
+ * {@link ExecutionGraph#stack()} will return concurrent "frames" of the graph
50
+ * in a topological sort order, meaning that nodes are always traversed before
51
+ * any of their successor nodes, and nodes of independent branches can be
52
+ * scheduled in parallel.
53
+ *
54
+ * For the same example graph:
55
+ *
56
+ * <pre><code>
57
+ * graph.stack().each { println it.join("|") }
58
+ * </code></pre>
59
+ *
60
+ * Would output:
61
+ *
62
+ * <pre>
63
+ * a|w|z
64
+ * b|x
65
+ * c|y
66
+ * d
67
+ * </pre>
68
+ */
69
+class ExecutionGraph implements Serializable {
70
+  /**
71
+   * Map of graph progression, nodes and their successor (out) nodes.
72
+   *
73
+   * @example
74
+   * An example graph and its <code>progression</code>.
75
+   *
76
+   * <pre><code>
77
+   *   a       w       z    [
78
+   *     ⇘   ⇙                a:[b],    w:[b],
79
+   *       b       x
80
+   *     ⇙   ⇘   ⇙            b:[c, y], x:[y],
81
+   *   c       y
82
+   *     ⇘   ⇙                c:[d],    y:[d],
83
+   *       d                ]
84
+   * </code></pre>
85
+   */
86
+  protected Map progression
87
+
88
+  /**
89
+   * Map of graph recession, nodes and their predecessor (in) nodes. Allows
90
+   * for efficient backward traversal.
91
+   *
92
+   * @example
93
+   * An example graph and its <code>recession</code>.
94
+   *
95
+   * <pre><code>
96
+   *   a       w       z    [
97
+   *     ⇘   ⇙                b:[a, w],
98
+   *       b       x
99
+   *     ⇙   ⇘   ⇙            c:[b],    y:[b, x],
100
+   *   c       y
101
+   *     ⇘   ⇙                d:[c, y],
102
+   *       d                ]
103
+   * </code></pre>
104
+   */
105
+  protected Map recession
106
+
107
+  /**
108
+   * Set of graph isolates, nodes that are unconnected from all other nodes.
109
+   */
110
+  protected Set isolates
111
+
112
+  /**
113
+   * Constructs a directed execution graph using the given sets of edge
114
+   * sequences (arcs).
115
+   *
116
+   * @example
117
+   * See {@link ExecutionGraph} for examples.
118
+   */
119
+  ExecutionGraph(List arcs) {
120
+    progression = [:]
121
+    recession = [:]
122
+    isolates = [] as Set
123
+
124
+    arcs.each { addArc(it as List) }
125
+  }
126
+
127
+  /**
128
+   * All ancestors of (nodes eventually leading to) the given node.
129
+   */
130
+  Set ancestorsOf(node) {
131
+    def parents = inTo(node)
132
+
133
+    parents + parents.inject([] as Set) { ancestors, parent -> ancestors + ancestorsOf(parent) }
134
+  }
135
+
136
+  /**
137
+   * Whether the given graph is equal to this one.
138
+   */
139
+  boolean equals(ExecutionGraph other) {
140
+    progression == other.progression && isolates == other.isolates
141
+  }
142
+
143
+  /**
144
+   * Returns all nodes that have no outgoing edges.
145
+   */
146
+  Set leaves() {
147
+    (recession.keySet() - progression.keySet()) + isolates
148
+  }
149
+
150
+  /**
151
+   * The number of nodes that lead directly to the given one.
152
+   */
153
+  int inDegreeOf(node) {
154
+    inTo(node).size()
155
+  }
156
+
157
+  /**
158
+   * The nodes that lead directly to the given one.
159
+   */
160
+  Set inTo(node) {
161
+    recession[node] ?: [] as Set
162
+  }
163
+
164
+  /**
165
+   * All nodes in the graph.
166
+   */
167
+  Set nodes() {
168
+    progression.keySet() + recession.keySet() + isolates
169
+  }
170
+
171
+  /**
172
+   * Returns a union of this graph and the given one.
173
+   */
174
+  ExecutionGraph or(ExecutionGraph other) {
175
+    def newGraph = new ExecutionGraph()
176
+
177
+    [this, other].each { source ->
178
+      source.progression.each { newGraph.addSuccession(it.key, it.value) }
179
+      source.isolates.each { newGraph.addIsolate(it) }
180
+    }
181
+
182
+    newGraph
183
+  }
184
+
185
+  /**
186
+   * The number of nodes the given one directly leads to.
187
+   */
188
+  int outDegreeOf(node) {
189
+    outOf(node).size()
190
+  }
191
+
192
+  /**
193
+   * The nodes the given one directly leads to.
194
+   */
195
+  Set outOf(node) {
196
+    progression[node] ?: [] as Set
197
+  }
198
+
199
+  /**
200
+   * Returns a concatenation of this graph and the given one.
201
+   */
202
+  ExecutionGraph plus(ExecutionGraph other) {
203
+    def newGraph = this | other
204
+
205
+    leaves().each { leaf ->
206
+      newGraph.addSuccession(leaf, other.roots())
207
+    }
208
+
209
+    newGraph
210
+  }
211
+
212
+  /**
213
+   * Returns all nodes that have no incoming edges.
214
+   */
215
+  Set roots() {
216
+    (progression.keySet() - recession.keySet()) + isolates
217
+  }
218
+
219
+  /**
220
+   * Returns each concurrent node "frames" of the graph in a topological sort
221
+   * order. See {@link ExecutionGraph} for examples. A {@link RuntimeException}
222
+   * will be thrown in the event a graph cycle is detected.
223
+   */
224
+  List stack() throws RuntimeException {
225
+    def concurrentFrames = []
226
+
227
+    def graphSize = (progression.keySet() + recession.keySet() + isolates).size()
228
+    def traversed = [] as Set
229
+    def prevNodes
230
+
231
+    while (traversed.size() < graphSize) {
232
+      def nextNodes
233
+
234
+      if (!prevNodes) {
235
+        nextNodes = roots()
236
+      } else {
237
+        nextNodes = [] as Set
238
+
239
+        prevNodes.each { prev ->
240
+          outOf(prev).each { outNode ->
241
+            if ((inTo(outNode) - traversed).isEmpty()) {
242
+              nextNodes.add(outNode)
243
+            }
244
+          }
245
+        }
246
+      }
247
+
248
+      if (!nextNodes && traversed.size() < graphSize) {
249
+        throw new RuntimeException("cycle detected in graph (${this})")
250
+      }
251
+
252
+      traversed.addAll(nextNodes)
253
+      prevNodes = nextNodes
254
+      concurrentFrames.add(nextNodes as List)
255
+    }
256
+
257
+    concurrentFrames
258
+  }
259
+
260
+  /**
261
+   * A string representation of the graph compatible with <code>dot</code>.
262
+   *
263
+   * @example
264
+   * Render the graph with dot
265
+   *
266
+   * <pre><code>
267
+   * $ echo "[graph.toString() value]" | dot -Tsvg &gt; graph.svg
268
+   * </code></pre>
269
+   */
270
+  String toString() {
271
+    def allEdges = progression.inject([]) { edges, predecessor, successors ->
272
+      edges + successors.collect { successor ->
273
+        "${predecessor} -> ${successor}"
274
+      }
275
+    }
276
+
277
+    'digraph { ' + (allEdges + isolates).join("; ") + ' }'
278
+  }
279
+
280
+  protected
281
+
282
+  /**
283
+   * Appends a new arc of nodes to the graph.
284
+   *
285
+   * @example
286
+   * An existing graph.
287
+   * <pre><code>
288
+   *   a
289
+   *     ⇘
290
+   *       b
291
+   *         ⇘
292
+   *           c
293
+   * </code></pre>
294
+   *
295
+   * Appended with <code>graph &lt;&lt; ["x", "b", "y", "z"]</code> becomes.
296
+   * <pre><code>
297
+   *   a       x
298
+   *     ⇘   ⇙
299
+   *       b
300
+   *     ⇙   ⇘
301
+   *   y       c
302
+   *     ⇘
303
+   *       z
304
+   * </code></pre>
305
+   */
306
+  @NonCPS
307
+  void addArc(List arc) {
308
+    if (arc.size() == 1) {
309
+      addIsolate(arc[0])
310
+    } else {
311
+      arc.eachWithIndex { node, i ->
312
+        if (i < (arc.size() - 1)) {
313
+          addSuccession(node, [arc[i+1]])
314
+        }
315
+      }
316
+    }
317
+  }
318
+
319
+  @NonCPS
320
+  void addIsolate(isolate) {
321
+    isolates.add(isolate)
322
+  }
323
+
324
+  @NonCPS
325
+  void addSuccession(predecessor, successors) {
326
+    if (!progression.containsKey(predecessor)) {
327
+      progression[predecessor] = [] as Set
328
+    }
329
+
330
+    progression[predecessor].addAll(successors)
331
+
332
+    successors.each { successor ->
333
+      if (!recession.containsKey(successor)) {
334
+        recession[successor] = [] as Set
335
+      }
336
+
337
+      recession[successor].add(predecessor)
338
+    }
339
+
340
+    isolates -= (successors + predecessor)
341
+  }
342
+}

+ 173
- 0
test/org/wikimedia/integration/ExecutionContextTest.groovy View File

@@ -0,0 +1,173 @@
1
+import groovy.mock.interceptor.MockFor
2
+import static groovy.test.GroovyAssert.*
3
+import groovy.util.GroovyTestCase
4
+
5
+import org.wikimedia.integration.ExecutionGraph
6
+import org.wikimedia.integration.ExecutionContext
7
+
8
+class ExecutionContextTest extends GroovyTestCase {
9
+  void testNodeContextBindings() {
10
+    /*
11
+     *  a
12
+     *    ⇘
13
+     *      b
14
+     *        ⇘
15
+     *          c       x
16
+     *            ⇘   ⇙
17
+     *              d
18
+     *            ⇙   ⇘
19
+     *          y       e
20
+     *            ⇘   ⇙
21
+     *              f
22
+     */
23
+    def context = new ExecutionContext(new ExecutionGraph([
24
+      ["a", "b", "c", "d", "e", "f"],
25
+      ["x", "d", "y", "f"],
26
+    ]))
27
+
28
+    def cContext = context.ofNode("c")
29
+    def eContext = context.ofNode("e")
30
+
31
+    context.ofNode("a").bind("foo", "afoo")
32
+    context.ofNode("a").bind("bar", "abar")
33
+    context.ofNode("b").bind("foo", "bfoo")
34
+    context.ofNode("x").bind("bar", "xbar")
35
+
36
+    assert cContext.binding("a", "foo") == "afoo"
37
+    assert cContext.binding("a", "bar") == "abar"
38
+    assert cContext.binding("b", "foo") == "bfoo"
39
+
40
+    shouldFail(ExecutionContext.AncestorNotFoundException) {
41
+      cContext.binding("x", "bar")
42
+    }
43
+
44
+    shouldFail(ExecutionContext.NameNotFoundException) {
45
+      cContext.binding("baz")
46
+    }
47
+
48
+    assert eContext.binding("a", "foo") == "afoo"
49
+    assert eContext.binding("a", "bar") == "abar"
50
+    assert eContext.binding("b", "foo") == "bfoo"
51
+    assert eContext.binding("x", "bar") == "xbar"
52
+  }
53
+
54
+  void testNodeContextBindings_immutable() {
55
+    /*
56
+     *  a
57
+     *    ⇘
58
+     *      b
59
+     *        ⇘
60
+     *          c
61
+     */
62
+    def context = new ExecutionContext(new ExecutionGraph([
63
+      ["a", "b", "c"],
64
+    ]))
65
+
66
+    context.ofNode("b").bind("foo", "bfoo")
67
+
68
+    shouldFail(ExecutionContext.NameAlreadyBoundException) {
69
+      context.ofNode("b").bind("foo", "newfoo")
70
+    }
71
+
72
+    assert context.ofNode("b").binding("foo") == "bfoo"
73
+  }
74
+
75
+  void testStringInterpolation() {
76
+    /*
77
+     *  a
78
+     *    ⇘
79
+     *      b
80
+     *        ⇘
81
+     *          c       x
82
+     *            ⇘   ⇙
83
+     *              d
84
+     *            ⇙   ⇘
85
+     *          y       e
86
+     *            ⇘   ⇙
87
+     *              f
88
+     */
89
+    def context = new ExecutionContext(new ExecutionGraph([
90
+      ["a", "b", "c", "d", "e", "f"],
91
+      ["x", "d", "y", "f"],
92
+    ]))
93
+
94
+    context.ofNode("a").bind("foo", "afoo")
95
+    context.ofNode("a").bind("bar", "abar")
96
+    context.ofNode("b").bind("foo", "bfoo")
97
+    context.ofNode("x").bind("bar", "xbar")
98
+
99
+    assert (context.ofNode("b") % 'w-t-${a.foo} ${.foo}') == "w-t-afoo bfoo"
100
+
101
+    assert (context.ofNode("c") % 'w-t-${a.foo} ${b.foo}') == "w-t-afoo bfoo"
102
+    assert (context.ofNode("c") % 'w-t-${a.bar}') == "w-t-abar"
103
+
104
+    shouldFail(ExecutionContext.AncestorNotFoundException) {
105
+      context.ofNode("x") % 'w-t-${b.bar}'
106
+    }
107
+
108
+    shouldFail(ExecutionContext.NameNotFoundException) {
109
+      context.ofNode("b") % 'w-t-${.bar}'
110
+    }
111
+  }
112
+
113
+  void testStringInterpolation_selfScope() {
114
+    /*
115
+     *  a
116
+     *    ⇘
117
+     *      b
118
+     *        ⇘
119
+     *          c
120
+     */
121
+    def context = new ExecutionContext(new ExecutionGraph([
122
+      ["a", "b", "c"],
123
+    ]))
124
+
125
+    context.ofNode("c").bind("foo", "cfoo")
126
+
127
+    assert (context.ofNode("c") % 'w-t-${.foo}') == "w-t-cfoo"
128
+  }
129
+
130
+  void testNodeContextGetAll() {
131
+    def context = new ExecutionContext(new ExecutionGraph([
132
+      ["a", "b", "c", "z"],
133
+      ["x", "b", "y", "z"],
134
+    ]))
135
+
136
+    context.ofNode("a").bind("foo", "afoo")
137
+    context.ofNode("c").bind("foo", "cfoo")
138
+    context.ofNode("y").bind("foo", "yfoo")
139
+
140
+    assert context.ofNode("z").getAll("foo") == ["afoo", "cfoo", "yfoo"]
141
+  }
142
+
143
+  void testNodeContextGetAt() {
144
+    def context = new ExecutionContext(new ExecutionGraph([
145
+      ["a", "b", "c"],
146
+    ]))
147
+
148
+    context.ofNode("a").bind("foo", "afoo")
149
+    context.ofNode("c").bind("foo", "cfoo")
150
+
151
+    assert context.ofNode("c")["a.foo"] == "afoo"
152
+    assert context.ofNode("c")["foo"] == "cfoo"
153
+  }
154
+
155
+  void testNodeContextGetAt_selfScope() {
156
+    /*
157
+     *  a
158
+     *    ⇘
159
+     *      b
160
+     *        ⇘
161
+     *          c
162
+     */
163
+    def context = new ExecutionContext(new ExecutionGraph([
164
+      ["a", "b", "c"],
165
+    ]))
166
+
167
+    def bContext = context.ofNode("b")
168
+
169
+    bContext["foo"] = "bfoo"
170
+
171
+    assert bContext[".foo"] == "bfoo"
172
+  }
173
+}

+ 178
- 0
test/org/wikimedia/integration/ExecutionGraphTest.groovy View File

@@ -0,0 +1,178 @@
1
+import groovy.mock.interceptor.MockFor
2
+import static groovy.test.GroovyAssert.*
3
+import groovy.util.GroovyTestCase
4
+
5
+import org.wikimedia.integration.ExecutionGraph
6
+
7
+class ExecutionGraphTest extends GroovyTestCase {
8
+  void testAncestorsOf() {
9
+    def graph = new ExecutionGraph([
10
+      ["a", "b", "c", "d", "e", "f"],
11
+      ["x", "d", "y", "f"],
12
+    ])
13
+
14
+    assert graph.ancestorsOf("c") == ["b", "a"] as Set
15
+    assert graph.ancestorsOf("d") == ["c", "b", "a", "x"] as Set
16
+    assert graph.ancestorsOf("f") == ["e", "d", "c", "b", "a", "x", "y"] as Set
17
+  }
18
+
19
+  void testLeaves() {
20
+    def graph = new ExecutionGraph([
21
+      ["a", "b", "c", "d"],
22
+      ["f", "b", "g"],
23
+    ])
24
+
25
+    assert graph.leaves() == ["g", "d"] as Set
26
+  }
27
+
28
+  void testInDegreeOf() {
29
+    def graph = new ExecutionGraph([
30
+      ["a", "b", "c"],
31
+      ["d", "b", "e"],
32
+      ["f", "b"],
33
+    ])
34
+
35
+    assert graph.inDegreeOf("a") == 0
36
+    assert graph.inDegreeOf("d") == 0
37
+    assert graph.inDegreeOf("f") == 0
38
+    assert graph.inDegreeOf("b") == 3
39
+    assert graph.inDegreeOf("e") == 1
40
+    assert graph.inDegreeOf("c") == 1
41
+  }
42
+
43
+  void testInTo() {
44
+    def graph = new ExecutionGraph([
45
+      ["a", "b", "c"],
46
+      ["d", "b", "e"],
47
+      ["f", "b"],
48
+    ])
49
+
50
+    assert graph.inTo("a") == [] as Set
51
+    assert graph.inTo("d") == [] as Set
52
+    assert graph.inTo("f") == [] as Set
53
+    assert graph.inTo("b") == ["a", "d", "f"] as Set
54
+    assert graph.inTo("c") == ["b"] as Set
55
+    assert graph.inTo("e") == ["b"] as Set
56
+  }
57
+
58
+  void testNodes() {
59
+    def graph = new ExecutionGraph([
60
+      ["a", "b", "c"],
61
+      ["x", "b", "y"],
62
+      ["z"],
63
+    ])
64
+
65
+    assert graph.nodes() == ["a", "b", "c", "x", "y", "z"] as Set
66
+  }
67
+
68
+  void testOr() {
69
+    def graph1 = new ExecutionGraph([
70
+      ["x", "y", "z"],
71
+    ])
72
+
73
+    def graph2 = new ExecutionGraph([
74
+      ["a", "b", "c"],
75
+      ["d", "b", "e"],
76
+    ])
77
+
78
+    assert (graph1 | graph2) == new ExecutionGraph([
79
+      ["x", "y", "z"],
80
+      ["a", "b", "c"],
81
+      ["d", "b", "e"],
82
+    ])
83
+  }
84
+
85
+  void testOutDegreeOf() {
86
+    def graph = new ExecutionGraph([
87
+      ["a", "b", "c"],
88
+      ["d", "b", "e"],
89
+      ["f", "b"],
90
+    ])
91
+
92
+    assert graph.outDegreeOf("a") == 1
93
+    assert graph.outDegreeOf("d") == 1
94
+    assert graph.outDegreeOf("f") == 1
95
+    assert graph.outDegreeOf("b") == 2
96
+    assert graph.outDegreeOf("e") == 0
97
+    assert graph.outDegreeOf("c") == 0
98
+  }
99
+
100
+  void testOutOf() {
101
+    def graph = new ExecutionGraph([
102
+      ["a", "b", "c"],
103
+      ["d", "b", "e"],
104
+      ["f", "b"],
105
+    ])
106
+
107
+    assert graph.outOf("a") == ["b"] as Set
108
+    assert graph.outOf("d") == ["b"] as Set
109
+    assert graph.outOf("f") == ["b"] as Set
110
+    assert graph.outOf("b") == ["c", "e"] as Set
111
+    assert graph.outOf("c") == [] as Set
112
+    assert graph.outOf("e") == [] as Set
113
+  }
114
+
115
+  void testPlus() {
116
+    def graph1 = new ExecutionGraph([
117
+      ["x", "y", "z"],
118
+    ])
119
+
120
+    def graph2 = new ExecutionGraph([
121
+      ["a", "b", "c"],
122
+      ["d", "b", "e"],
123
+    ])
124
+
125
+    assert (graph1 + graph2) == new ExecutionGraph([
126
+      ["x", "y", "z", "a", "b", "c"],
127
+      ["x", "y", "z", "d", "b", "e"],
128
+    ])
129
+  }
130
+
131
+  void testRoots() {
132
+    def graph = new ExecutionGraph([
133
+      ["a", "b", "c", "d", "e"],
134
+      ["f", "b", "g", "e"],
135
+    ])
136
+
137
+    assert graph.roots() == ["a", "f"] as Set
138
+  }
139
+
140
+  void testRootsOfIsolates() {
141
+    def graph = new ExecutionGraph([["a"], ["z"]])
142
+
143
+    assert graph.roots() == ["a", "z"] as Set
144
+  }
145
+
146
+  void testStack() {
147
+    def graph = new ExecutionGraph([
148
+      ["a", "b", "c", "d", "e", "f"],
149
+      ["x", "d", "y", "f"],
150
+    ])
151
+
152
+    assert graph.stack() == [
153
+      ["a", "x"],
154
+      ["b"],
155
+      ["c"],
156
+      ["d"],
157
+      ["e", "y"],
158
+      ["f"],
159
+    ]
160
+  }
161
+
162
+  void testStack_cycleDetection() {
163
+    def graph = new ExecutionGraph([
164
+      ["a", "b", "c"],
165
+      ["x", "b", "y", "a"],
166
+    ])
167
+
168
+    shouldFail(Exception) {
169
+      graph.stack()
170
+    }
171
+  }
172
+
173
+  void testToString() {
174
+    def graph = new ExecutionGraph([["a", "b", "c"], ["x"]])
175
+
176
+    assert graph.toString() == "digraph { a -> b; b -> c; x }"
177
+  }
178
+}

Loading…
Cancel
Save