You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

326 lines
9.1 KiB

pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
  1. import groovy.mock.interceptor.MockFor
  2. import static groovy.test.GroovyAssert.*
  3. import groovy.util.GroovyTestCase
  4. import java.io.FileNotFoundException
  5. import org.wikimedia.integration.Blubber
  6. import org.wikimedia.integration.PipelineRunner
  7. import org.wikimedia.integration.Utility
  8. class PipelineRunnerTest extends GroovyTestCase {
  9. private class WorkflowScript {} // Mock for Jenkins Pipeline workflow context
  10. void setUp() {
  11. // Mock all static calls to Utility.randomAlphanum
  12. Utility.metaClass.static.randomAlphanum = { "randomfoo" }
  13. }
  14. void tearDown() {
  15. // Reset static mocks
  16. Utility.metaClass = null
  17. }
  18. void testConstructor_workflowScript() {
  19. new PipelineRunner(new WorkflowScript())
  20. }
  21. void testConstructor_workflowScriptAndProperties() {
  22. new PipelineRunner(new WorkflowScript(), configPath: "foo")
  23. }
  24. void testGetConfigFile() {
  25. def pipeline = new PipelineRunner(new WorkflowScript(), configPath: "foo")
  26. assert pipeline.getConfigFile("bar") == "foo/bar"
  27. }
  28. void testGetTempFile() {
  29. def pipeline = new PipelineRunner(new WorkflowScript(), configPath: "foo")
  30. assert pipeline.getTempFile("bar") ==~ /^foo\/bar[a-z0-9]+$/
  31. }
  32. void testQualifyRegistryPath() {
  33. def pipeline = new PipelineRunner(new WorkflowScript())
  34. def url = pipeline.qualifyRegistryPath("foo")
  35. assert url == "docker-registry.wikimedia.org/wikimedia/foo"
  36. }
  37. void testQualifyRegistryPath_disallowsSlashes() {
  38. def pipeline = new PipelineRunner(new WorkflowScript())
  39. shouldFail(AssertionError) {
  40. pipeline.qualifyRegistryPath("foo/bar")
  41. }
  42. }
  43. void testBuild_checksWhetherConfigExists() {
  44. def mockWorkflow = new MockFor(WorkflowScript)
  45. mockWorkflow.demand.fileExists { path ->
  46. assert path == ".pipeline/nonexistent.yaml"
  47. false
  48. }
  49. mockWorkflow.use {
  50. def runner = new PipelineRunner(new WorkflowScript(),
  51. configPath: ".pipeline",
  52. blubberConfig: "nonexistent.yaml")
  53. shouldFail(FileNotFoundException) {
  54. runner.build("foo", [bar: "baz"])
  55. }
  56. }
  57. }
  58. void testBuild_generatesDockerfileAndBuilds() {
  59. def mockWorkflow = new MockFor(WorkflowScript)
  60. def mockBlubber = new MockFor(Blubber)
  61. mockWorkflow.demand.fileExists { true }
  62. mockBlubber.demand.generateDockerfile { variant ->
  63. assert variant == "foo"
  64. "BASE: foo\n"
  65. }
  66. mockWorkflow.demand.writeFile { args ->
  67. assert args.text == "BASE: foo\n"
  68. assert args.file ==~ /^\.pipeline\/Dockerfile\.[a-z0-9]+$/
  69. }
  70. mockWorkflow.demand.sh { args ->
  71. assert args.returnStdout
  72. assert args.script ==~ (/^docker build --pull --label 'foo=a' --label 'bar=b' / +
  73. /--file '\.pipeline\/Dockerfile\.[a-z0-9]+' \.$/)
  74. // Mock `docker build` output to test that we correctly parse the image ID
  75. return "Removing intermediate container foo\n" +
  76. " ---> bf1e86190382\n" +
  77. "Successfully built bf1e86190382\n"
  78. }
  79. mockWorkflow.use {
  80. mockBlubber.use {
  81. def runner = new PipelineRunner(new WorkflowScript())
  82. assert runner.build("foo", [foo: "a", bar: "b"]) == "bf1e86190382"
  83. }
  84. }
  85. }
  86. void testDeploy_checksConfigForChart() {
  87. def mockWorkflow = new MockFor(WorkflowScript)
  88. mockWorkflow.demand.readYaml { Map args ->
  89. assert args.file == ".foo/bar.yaml"
  90. [chart: null]
  91. }
  92. mockWorkflow.use {
  93. def runner = new PipelineRunner(new WorkflowScript(),
  94. configPath: ".foo",
  95. helmConfig: "bar.yaml")
  96. shouldFail(AssertionError) {
  97. runner.deploy("foo/name", "footag")
  98. }
  99. }
  100. }
  101. void testDeploy_executesHelm() {
  102. def mockWorkflow = new MockFor(WorkflowScript)
  103. mockWorkflow.demand.readYaml {
  104. [chart: "http://an.example/chart.tgz"]
  105. }
  106. mockWorkflow.demand.sh { cmd ->
  107. def expectedCmd = "helm --tiller-namespace='ci' install " +
  108. "--namespace='ci' --set " +
  109. "'docker.registry=docker-registry.wikimedia.org'," +
  110. "'docker.pull_policy=IfNotPresent'," +
  111. "'main_app.image=wikimedia/foo/name'," +
  112. "'main_app.version=footag' " +
  113. "-n 'foo/name-randomfoo' " +
  114. "--debug --wait --timeout 120 " +
  115. "'http://an.example/chart.tgz'"
  116. assert cmd == expectedCmd
  117. }
  118. mockWorkflow.use {
  119. def runner = new PipelineRunner(new WorkflowScript())
  120. runner.deploy("foo/name", "footag")
  121. }
  122. }
  123. void testDeploy_executesHelmWithKubeConfig() {
  124. def mockWorkflow = new MockFor(WorkflowScript)
  125. mockWorkflow.demand.readYaml {
  126. [chart: "http://an.example/chart.tgz"]
  127. }
  128. mockWorkflow.demand.sh { cmd ->
  129. def expectedCmd = "KUBECONFIG='/etc/kubernetes/foo.config' " +
  130. "helm --tiller-namespace='ci' install " +
  131. "--namespace='ci' --set " +
  132. "'docker.registry=docker-registry.wikimedia.org'," +
  133. "'docker.pull_policy=IfNotPresent'," +
  134. "'main_app.image=wikimedia/foo/name'," +
  135. "'main_app.version=footag' " +
  136. "-n 'foo/name-randomfoo' " +
  137. "--debug --wait --timeout 120 " +
  138. "'http://an.example/chart.tgz'"
  139. assert cmd == expectedCmd
  140. }
  141. mockWorkflow.use {
  142. def runner = new PipelineRunner(new WorkflowScript(),
  143. kubeConfig: "/etc/kubernetes/foo.config")
  144. runner.deploy("foo/name", "footag")
  145. }
  146. }
  147. void testPurgeRelease_executesHelm() {
  148. def mockWorkflow = new MockFor(WorkflowScript)
  149. mockWorkflow.demand.sh { cmd ->
  150. def expectedCmd = "helm --tiller-namespace='ci' delete --purge 'foorelease'"
  151. assert cmd == expectedCmd
  152. }
  153. mockWorkflow.use {
  154. def runner = new PipelineRunner(new WorkflowScript())
  155. runner.purgeRelease("foorelease")
  156. }
  157. }
  158. void testPurgeRelease_executesHelmWithKubeConfig() {
  159. def mockWorkflow = new MockFor(WorkflowScript)
  160. mockWorkflow.demand.sh { cmd ->
  161. def expectedCmd = "KUBECONFIG='/etc/kubernetes/foo.config' " +
  162. "helm --tiller-namespace='ci' delete --purge 'foorelease'"
  163. assert cmd == expectedCmd
  164. }
  165. mockWorkflow.use {
  166. def runner = new PipelineRunner(new WorkflowScript(),
  167. kubeConfig: "/etc/kubernetes/foo.config")
  168. runner.purgeRelease("foorelease")
  169. }
  170. }
  171. void testTestRelease_executesHelm() {
  172. def mockWorkflow = new MockFor(WorkflowScript)
  173. mockWorkflow.demand.sh { cmd ->
  174. def expectedCmd = "helm --tiller-namespace='ci' test --cleanup 'foorelease'"
  175. assert cmd == expectedCmd
  176. }
  177. mockWorkflow.use {
  178. def runner = new PipelineRunner(new WorkflowScript())
  179. runner.testRelease("foorelease")
  180. }
  181. }
  182. void testTestRelease_executesHelmWithKubeConfig() {
  183. def mockWorkflow = new MockFor(WorkflowScript)
  184. mockWorkflow.demand.sh { cmd ->
  185. def expectedCmd = "KUBECONFIG='/etc/kubernetes/foo.config' " +
  186. "helm --tiller-namespace='ci' test --cleanup 'foorelease'"
  187. assert cmd == expectedCmd
  188. }
  189. mockWorkflow.use {
  190. def runner = new PipelineRunner(new WorkflowScript(),
  191. kubeConfig: "/etc/kubernetes/foo.config")
  192. runner.testRelease("foorelease")
  193. }
  194. }
  195. void testRegisterAs() {
  196. def mockWorkflow = new MockFor(WorkflowScript)
  197. mockWorkflow.demand.sh { cmd ->
  198. assert cmd == "docker tag 'fooID' 'internal.example/foorepo/fooname:footag' && " +
  199. "sudo /usr/local/bin/docker-pusher 'internal.example/foorepo/fooname:footag'"
  200. }
  201. mockWorkflow.use {
  202. def runner = new PipelineRunner(new WorkflowScript(),
  203. registryInternal: 'internal.example',
  204. repository: 'foorepo')
  205. runner.registerAs("fooID", "fooname", "footag")
  206. }
  207. }
  208. void testRegisterAs_bailsOnSlashes() {
  209. def mockWorkflow = new MockFor(WorkflowScript)
  210. mockWorkflow.use {
  211. def runner = new PipelineRunner(new WorkflowScript())
  212. shouldFail(AssertionError) {
  213. runner.registerAs("fooID", "foo/name", "footag")
  214. }
  215. }
  216. }
  217. void testRemoveImage() {
  218. def mockWorkflow = new MockFor(WorkflowScript)
  219. mockWorkflow.demand.sh { cmd ->
  220. assert cmd == "docker rmi --force 'fooID'"
  221. }
  222. mockWorkflow.use {
  223. def runner = new PipelineRunner(new WorkflowScript())
  224. runner.removeImage("fooID")
  225. }
  226. }
  227. void testRun() {
  228. def mockWorkflow = new MockFor(WorkflowScript)
  229. mockWorkflow.demand.timeout { Map args, Closure c ->
  230. assert args.time == 20
  231. assert args.unit == "MINUTES"
  232. c()
  233. }
  234. mockWorkflow.demand.sh { cmd ->
  235. assert cmd == "exec docker run --rm 'foo'"
  236. }
  237. mockWorkflow.use {
  238. def runner = new PipelineRunner(new WorkflowScript())
  239. runner.run("foo")
  240. }
  241. }
  242. }