Refactored `copies` configuration to allow for greater control over when
and which both local build context files and variant artifacts are
copied into the target image.
The new configuration introduces a "local" keyword to signify when a
`copies` entry should apply to the files from the local build context as
opposed to files from another variant's image during a multi-stage
build.
variants:
build:
copies:
- from: local
source: ./src
destination: .
Note that with this change, the user must now explicitly define whether
_any_ files should be copied in from the local build context. None will
be copied in by default.
To help keep configurations succinct, especially considering this new
requirement, and to approximate the old `copies: variant`, a shorthand
format and sane defaults for `source` and `destination` (depending whether
`from` is "local" or a variant name) were implemented.
variants:
build:
copies: [local]
development:
copies:
- from: build
- from: local
source: ./config.dev.yaml
destination: ./config.yaml
The shorthand:
copies: [ref, ...]
# is equivalent to
copies: [{ from: ref }, ...]
And the following defaults are used when no `source` and `destination`
are specified.
copies:
- from: local
# defaults to
- from: local
source: .
destination: .
copies:
- from: variant
# defaults to two entries
- from: variant
source: /srv/app # the lives.in dir
destination: /srv/app
- from: variant
source: /opt/local # the shared lib dir
destination: /opt/local
Bug: T211625
Change-Id: I4c4217905afc0762b6bd66ed594d43cc0486e3e2
Uses the github.com/ghodss/yaml library to convert YAML to JSON before
unmarshaling for the purposes of supporting YAML and JSON input while
converting to only support JSON internally.
Bug: T207694
Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
Support for mounting a shared volume instead of copying application
files was meant to provide an option for development use cases. This
functionality has never been thoroughly tested or utilized for any use
case. It should be removed for now.
Relates tangentially to task T204591 that simplifies node support.
Change-Id: Ib70cb7bceb504841897a38d732880ba376fe67c8
All import paths and other references to the previously Blubber repo in
Phabricator have been changed to point to the new Gerrit project URL.
Note that building or testing against this change will require you to
move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
Contribution documentation was updated to reflect the changes, and the
`.arcvendor` submodule has been removed.
Bug: T200452
Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
Summary:
Use cases involving running of test suites and doc generation require
more liberal ownership and read/write permission to application files.
When `runs.insecurely` is set to `true`, the effective runtime user will
be `lives.as`, the same user that owns the application files and
installed dependencies.
D999 is a complement to this change to allow restrictions on this and
other potentially sensitive configuration.
Depends on D999, D1002
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1003
Summary:
The implementation of D984 did not include enforcing ownership for
`build.CopyFrom` instruction and so artifacts copied from one image to
another via `copies:` were problematically owned as root.
In order to fix this behavior:
1. `config.ArtifactConfig` `build.CopyFrom` instructions are now
injected duration `build.PhaseInstall`
2. `config.VariantConfig` calls `build.ApplyUser` for these artifact
instructions as well using the `runs.as` user
3. `build.CopyAs` was refactored to wrap any `build.Instruction` which
should only really be used with `build.Copy` or `build.CopyFrom`.
Test Plan:
Run `go test ./...`. Run `blubber` against configuration with a variant that
uses `copies` and verify that the `COPY --from` instructions also include a
`--chown` flag.
Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1002
Summary:
Introduce new `build.EntryPoint` and `build.WorkingDirectory`
instructions to allow configuration to inject them instead of hard
coding their generation in the Docker compiler.
Simplified the Docker compiler to simply iterate over build phases as
returned by a new function `build.Phases()`.
Depends on D990
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D991
Summary:
Introduces new `lives` configuration that provides the name/UID/GID of
the user that will own application files and installed dependencies.
This new configuration is distinct from `runs` in that the former
determines application file location ownership and the latter now only
determines runtime process ownership. Default configuration has also
been introduced for both config sections.
In addition to the new configuration, a new `build.CopyAs` instruction
has been introduced that ensures correct UID/GID ownership of files
copied into the container image, and all unqualified `build.Copy`
instructions are wrapped by the new `build.CopyAs` instruction using the
UID/GID appropriate for the current build phase. A new `build.User`
instruction is also introduced and injected into the build at the start
of certain phases to enforce ownership of `build.Run` processes.
This effective process/file ownership model is:
PhasePrivileged - "root"
PhasePrivilegedDropped - lives.as
PhasePreInstall - lives.as
PhaseInstall - lives.as
PhasePostInstall - runs.as
Fixes T187372
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Subscribers: mmodell
Tags: #release-engineering-team
Maniphest Tasks: T187372
Differential Revision: https://phabricator.wikimedia.org/D984
Summary:
Implemented a validation system using the
`github.com/go-playground/validator` package, extending it with custom
validation tags, and implemented translation of validation errors into
somewhat human-friendly messages.
Fixes T175186
Depends on D845
Test Plan: Run the unit tests and try running blubber against some bad config.
Reviewers: thcipriani, hashar, Jrbranaa, Joe, #release-engineering-team, mobrovac
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175186
Differential Revision: https://phabricator.wikimedia.org/D868
Summary:
Wrote inline documentation for all the things.
Fixes T168000
Test Plan: Execute `godoc -http=:6060` and proofread all the things.
Reviewers: thcipriani, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team, demon
Tags: #release-engineering-team
Maniphest Tasks: T168000
Differential Revision: https://phabricator.wikimedia.org/D841
Summary:
Now that our Phabricator instance handles `go get` requests, we can use
a path without the `.git` suffix as the canonical import path.
Test Plan: Run `go test ./...` and `go build`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D780
Summary:
Defined new abstract `build.Volume` and corresponding
`docker.DockerVolume` instructions.
Refactored compilation of main `COPY` or `VOLUME` instruction for
application files to use the new instructions and moved injection of
these instructions out of the compiler and into `VariantConfig`. The
latter can be smarter about the following cases:
1. When `copies` is set, simply depend on artifacts for the application
files and do not copy anything from the build host.
2. When `sharedvolume` is `true`, inject a `build.Volume` instruction
for the application working directory.
3. When neither of the above are set, copy application files from the
host.
Fixes T174623
Depends on D768
Test Plan:
Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure:
1. The `prep` stage has a `COPY . .` instruction.
2. The final stage has no `COPY . .` instruction, only `COPY --from=prep`
instructions.
Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174623
Differential Revision: https://phabricator.wikimedia.org/D769
Summary:
Support a `copies` variant config entry that will result in a
multi-stage build, copying both shared library files and application
directory from a previously defined variant. This is essentially a
shorthand for two `artifacts` entries that are likely to be idiomatic to
multi-stage build/prod configurations.
Defined a new abstract `build.CopyFrom` instruction and corresponding
`docker.DockerCopyFrom` instruction and refactored the writing of these
Dockerfile lines to be accomplished using an `InstructionsForPhase`
method on `config.ArtifactsConfig`.
Implemented new support for `copies` configuration in
`config.VariantConfig` and an `InstructionsForPhase` method that returns
`build.CopyFrom` instructions for both the shared library and
application directories.
Fixes T174622
Depends on D759
Test Plan:
Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure
the right `COPY --from` lines are included for the final stage.
Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174622
Differential Revision: https://phabricator.wikimedia.org/D768
Summary:
Establish `/opt/lib` as the location for installing application
dependencies that are installed via unprivileged execution and from
untrusted sources. The directory is created during the privileged build
phase and owned by the unprivileged runtime user.
Depends on D741
Test Plan: Run `go test ./...` or `arc unit`.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, mmodell, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D756
Summary:
One of Golang's "advantages" is not quibbling over style. To this end
the `go fmt` command exists. This is the result of me running:
find . -name '*.go' -exec go fmt {} \;
Test Plan: built package, ran tests
Reviewers: dduvall, mmodell, #release-engineering-team
Reviewed By: dduvall, mmodell, #release-engineering-team
Subscribers: hashar
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D694
Reorganized config type declarations.
Added `RunConfig` for declaring working directory and unprivileged
runtime user. A system for enforcing restrictions should probably be
implemented for this.
Implemented Dockerfile compiler, supporting multi-stage output in cases
whether artifacts from other variants are configured.
Dan Duvall
Tyler Cipriani