We've changed the config to be more consistent about "copies" and
"artifacts". This means that "copies" no longer accepts a string, but a
slice of strings. We should bump the config version and blubber version
to make it clear that we're not longer backwards compatible.
Change-Id: Ib68f623cbdcdf0aed74f4fdaec062835748e8eb9
Uses the github.com/ghodss/yaml library to convert YAML to JSON before
unmarshaling for the purposes of supporting YAML and JSON input while
converting to only support JSON internally.
Bug: T207694
Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
All import paths and other references to the previously Blubber repo in
Phabricator have been changed to point to the new Gerrit project URL.
Note that building or testing against this change will require you to
move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
Contribution documentation was updated to reflect the changes, and the
`.arcvendor` submodule has been removed.
Bug: T200452
Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
Summary:
Since `config.Validate` was changed to take any interface as an
argument, many of the validation tests can be refactored in a way that
avoids having to parse the full config context in YAML and instead
validates each specific config struct directly. The new test pattern is
simpler and less prone to future breakage should unrelated parts of the
overall config change.
Tests that rely on root config context were left unchanged.
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1023
Summary:
Introduced a `version` config field that must be specified and match
`config.CurrentVersion`.
Changed `config.ReadConfig` to use `yaml.UnmarshalStrict` to ensure that
errors are surfaced when unknown/bad fields are present in the given
YAML config. A smaller `config.VersionConfig` is now unmarshaled first
to prevalidate the new `version` field before the entire config is
parsed.
Fixes T191460
Test Plan:
Run `go test ./...`. Run `blubber` against some configuration containing
invalid fields and ensure that it surfaces a YAML error.
Reviewers: thcipriani, demon, hashar, mmodell, mobrovac, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T191460
Differential Revision: https://phabricator.wikimedia.org/D1021
Summary:
Use cases involving running of test suites and doc generation require
more liberal ownership and read/write permission to application files.
When `runs.insecurely` is set to `true`, the effective runtime user will
be `lives.as`, the same user that owns the application files and
installed dependencies.
D999 is a complement to this change to allow restrictions on this and
other potentially sensitive configuration.
Depends on D999, D1002
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1003
Summary:
Introduces new `lives` configuration that provides the name/UID/GID of
the user that will own application files and installed dependencies.
This new configuration is distinct from `runs` in that the former
determines application file location ownership and the latter now only
determines runtime process ownership. Default configuration has also
been introduced for both config sections.
In addition to the new configuration, a new `build.CopyAs` instruction
has been introduced that ensures correct UID/GID ownership of files
copied into the container image, and all unqualified `build.Copy`
instructions are wrapped by the new `build.CopyAs` instruction using the
UID/GID appropriate for the current build phase. A new `build.User`
instruction is also introduced and injected into the build at the start
of certain phases to enforce ownership of `build.Run` processes.
This effective process/file ownership model is:
PhasePrivileged - "root"
PhasePrivilegedDropped - lives.as
PhasePreInstall - lives.as
PhaseInstall - lives.as
PhasePostInstall - runs.as
Fixes T187372
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Subscribers: mmodell
Tags: #release-engineering-team
Maniphest Tasks: T187372
Differential Revision: https://phabricator.wikimedia.org/D984
Summary:
Implemented a validation system using the
`github.com/go-playground/validator` package, extending it with custom
validation tags, and implemented translation of validation errors into
somewhat human-friendly messages.
Fixes T175186
Depends on D845
Test Plan: Run the unit tests and try running blubber against some bad config.
Reviewers: thcipriani, hashar, Jrbranaa, Joe, #release-engineering-team, mobrovac
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175186
Differential Revision: https://phabricator.wikimedia.org/D868
Summary:
Fixed all linter warnings and advice except for vet's rule about unkeyed
composite literals which was disabled via a `-composites=false` flag in
`.arclint`. Most unkeyed literals (e.g. `build.Run{"command"}`) in this
project just seem too usefully succinct compared to their more verbose
keyed counterparts.
Depends on D841
Test Plan: Run `arc lint --everything` and verify there are no warnings or advice.
Reviewers: thcipriani, hashar, #release-engineering-team
Reviewed By: thcipriani, hashar, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D845
Summary:
Debian only provides a package for the GitHub version so it's either
this or we'd have to maintain a patch to change the import paths at
packaging, a moving target for sure.
Since it's only a development dependency, we're not too worried about
versioning it and this is a much more simple solution.
Refs T175609
Test Plan: Run `go get ./... && go test ./...`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175609
Differential Revision: https://phabricator.wikimedia.org/D782
Summary:
Now that our Phabricator instance handles `go get` requests, we can use
a path without the `.git` suffix as the canonical import path.
Test Plan: Run `go test ./...` and `go build`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D780
Summary:
Establish `/opt/lib` as the location for installing application
dependencies that are installed via unprivileged execution and from
untrusted sources. The directory is created during the privileged build
phase and owned by the unprivileged runtime user.
Depends on D741
Test Plan: Run `go test ./...` or `arc unit`.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, mmodell, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D756
Summary:
Refactored build instructions to use concrete types and
`build.Instruction` as an interface instead of relying on a simple enum
and arbitrary string arguments. The formal types result in:
1. Clearer internal data structures
2. Partial compilation and proper argument quoting for all instructions
moved into the common `build` package
3. Higher order instructions like `build.RunAll` that easily reduce to
compiler specific output
Test Plan: Run `arc unit` or `go test ./...`
Reviewers: thcipriani, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D741
Summary:
Refactored build instructions in `RunsConfig` to properly quote command
arguments injected from user data.
Established unit tests for `RunsConfig` instruction phases.
Refs T170285. Depends on D711
Omit ENV instruction when `runs.environment` is empty
Fixes T170285
Test Plan:
Run `arc unit`. Create a config without `runs.environment` defined and verify
that no bare `ENV` line ends up in the `Dockerfile` output.
Reviewers: thcipriani, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T170285
Differential Revision: https://phabricator.wikimedia.org/D715
Summary:
Added support for definition of environment variables under
`runs.environment`. Corresponding `ENV` instructions will be added to
Dockerfile output for the unprivileged build phase.
Fixes T168425
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Maniphest Tasks: T168425
Differential Revision: https://phabricator.wikimedia.org/D691
Tyler Cipriani
Dan Duvall